Learning non-basic conda project structures /u/CappinAndLion Python Education

I’m a longtime software developer (js, ror, .net, blah blah blah) getting in to some data science for a project I’m working on. I want to do specifically some text and data analysis. I’ve been using conda and creating some basic files that do operations for me ex. take a .csv and process the values and output some data in another file.

I want to take this to the next step and was looking for some guidance –

1. Be able to run each .py file as a job taking in parameters

2. Have queues for the above

3. Have some shared operations that each job can use

In RoR I would use Sidekiq to manage queues and kick off jobs etc… Are there any project recommendations for how to accomplish the above?

I’m fine with using cron but I’d imagine this has been codified in the past with thought.

I’m a longtime software developer (js, ror, .net, blah blah blah) getting in to some data science for a project I’m working on. I want to do specifically some text and data analysis. I’ve been using conda and creating some basic files that do operations for me ex. take a .csv and process the values and output some data in another file.

I want to take this to the next step and was looking for some guidance –

1. Be able to run each .py file as a job taking in parameters

2. Have queues for the above

3. Have some shared operations that each job can use

In RoR I would use Sidekiq to manage queues and kick off jobs etc… Are there any project recommendations for how to accomplish the above?

I’m fine with using cron but I’d imagine this has been codified in the past with thought.