For what it's worth, Shuttle was originally written for Sidekiq, not Sidekiq Pro. In order to accomplish the job dependency features that Shuttle requires, I wrote my own Sidekiq-based job chaining and dependency system. Now I am apparently not as good at concurrency as @mperham, so the code didn't work very well and would often fail in weird ways. This is why we ultimately switched to Sidekiq Pro.

The original commit moving from Sidekiq to Pro is not in the open-source repo, but it is in the Square internal repo. I've attached it here so that you can use it as a starting point in creating a non-pro version of the code, if you wish.

commit 78aaa71c1fef4ae125e17487973f1dfde8b8da1a
Author: Tim Morgan <tim@squareup.com>
Date:   Thu Mar 20 13:36:03 2014 -0700

    Replace SidekiqWorkerTracking with Sidekiq Pro batches feature

    * Workers relating to importing a commit are now all grouped under the same
      batch.
    * Because Sidekiq batches does not support nesting, we cannot have BlobImporters
      batch up KeyCreators, and CommitImporters batch up BlobImporters. Therefore,
      all three of these job types are under one batch. When that batch finishes,
      the Commit is marked as not loading _and_ all its blobs are marked as not
      loading.
      * To facilitate this, a new association has been introduced, `blobs_commits`,
        that associates a Commit with its Blobs.
    * Operations relating to counting and listing workers have been altered to use
      `Sidekiq::Batch::Stats`.
    * The ability to clear workers has been removed. This doesn't seem to exist
      in Sidekiq Pro.

diff --git a/app/controllers/commits_controller.rb b/app/controllers/commits_controller.rb
index 4fa6236..5925782 100644
--- a/app/controllers/commits_controller.rb
+++ b/app/controllers/commits_controller.rb
@@ -21,13 +21,12 @@ class CommitsController < ApplicationController

   before_filter :authenticate_user!, except: [:manifest, :localize]
   before_filter :monitor_required, except: [:show, :manifest, :localize]
-  before_filter :admin_required, only: :clear

   before_filter :find_project
   before_filter :find_commit, except: [:create, :manifest, :localize]

   respond_to :html, :json, only: [:show, :create, :update, :destroy, :import,
-                                  :sync, :match, :redo, :clear, :recalculate]
+                                  :sync, :match, :redo, :recalculate]

   # Renders JSON information about a Commit and its translation progress.
   #
@@ -175,7 +174,9 @@ class CommitsController < ApplicationController
   # | `locale` | The RFC 5646 identifier for a locale. |

   def import
-    CommitImporter.perform_once @commit.id, locale: params[:locale]
+    @commit.import_batch.jobs do
+      CommitImporter.perform_once @commit.id, locale: params[:locale]
+    end
     respond_with @commit, location: nil
   end

@@ -196,7 +197,9 @@ class CommitsController < ApplicationController
   # | `id`         | The SHA of a Commit.   |

   def sync
-    CommitImporter.perform_once @commit.id
+    @commit.import_batch.jobs do
+      CommitImporter.perform_once @commit.id
+    end
     respond_with @commit, location: nil
   end

@@ -218,30 +221,9 @@ class CommitsController < ApplicationController
   # | `id`         | The SHA of a Commit.   |

   def redo
-    @commit.update_attribute(:loading, true)
-    CommitImporter.perform_once @commit.id, force: true
-    respond_with @commit, location: project_commit_url(@project, @commit)
-  end
-
-  # Removes all workers from the loading list, marks the Commit as not loading,
-  # and recalculates Commit statistics if the Commit was previously loading.
-  # This method should be used to fix "stuck" Commits.
-  #
-  # Routes
-  # ------
-  #
-  # * `POST /projects/:project_id/commits/:id/clear`
-  #
-  # Path Parameters
-  # ---------------
-  #
-  # |              |                        |
-  # |:-------------|:-----------------------|
-  # | `project_id` | The slug of a Project. |
-  # | `id`         | The SHA of a Commit.   |
-
-  def clear
-    @commit.clear_workers!
+    @commit.import_batch.jobs do
+      CommitImporter.perform_once @commit.id, force: true
+    end
     respond_with @commit, location: project_commit_url(@project, @commit)
   end

diff --git a/app/models/blob.rb b/app/models/blob.rb
index 6c8d58e..c04d7df 100644
--- a/app/models/blob.rb
+++ b/app/models/blob.rb
@@ -23,6 +23,7 @@
 # |:----------|:-----------------------------------------------------|
 # | `project` | The {Project} whose repository this blob belongs to. |
 # | `keys`    | The {Key Keys} found in this blob.                   |
+# | `commits` | The {Commit Commits} with this blob.                 |
 #
 # Fields
 # ======
@@ -33,13 +34,13 @@
 # | `loading` | If `true`, one or more workers is currently importing {Key Keys} for this blob. This defaults to `true` to ensure that a Blob's Keys are loaded at least once before the cached result is used. |

 class Blob < ActiveRecord::Base
-  include SidekiqWorkerTracking
-
   self.primary_keys = :project_id, :sha_raw

   belongs_to :project, inverse_of: :blobs
   has_many :blobs_keys, foreign_key: [:project_id, :sha_raw], inverse_of: :blob, dependent: :delete_all
   has_many :keys, through: :blobs_keys
+  has_many :blobs_commits, foreign_key: [:project_id, :sha_raw], inverse_of: :blob, dependent: :delete_all
+  has_many :commits, through: :blobs_commits

   extend GitObjectField
   git_object_field :sha,
@@ -97,6 +98,7 @@ class Blob < ActiveRecord::Base
   # @private
   def inspect(default_behavior=false)
     return super() if default_behavior
-    "#<#{self.class.to_s} #{sha}>"
+    state = loading? ? 'loading' : 'cached'
+    "#<#{self.class.to_s} #{sha} (#{state})>"
   end
 end
diff --git a/app/models/blobs_commit.rb b/app/models/blobs_commit.rb
new file mode 100644
index 0000000..b3432d4
--- /dev/null
+++ b/app/models/blobs_commit.rb
@@ -0,0 +1,29 @@
+# Copyright 2013 Square Inc.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+# Join table between {Blob} and {Commit}. Indicates what blobs are accessible
+# from what commits.
+#
+# Associations
+# ------------
+#
+# |          |                                 |
+# |:---------|:--------------------------------|
+# | `blob`   | The {Blob} found in the Commit. |
+# | `commit` | The {Commit} with the Blob.     |
+
+class BlobsCommit < ActiveRecord::Base
+  belongs_to :blob, foreign_key: [:project_id, :sha_raw], inverse_of: :blobs_commits
+  belongs_to :commit, inverse_of: :blobs_commits
+end
diff --git a/app/models/commit.rb b/app/models/commit.rb
index 2bab2dd..cd1d379 100644
--- a/app/models/commit.rb
+++ b/app/models/commit.rb
@@ -45,6 +45,7 @@ require 'fileutils'
 # | `user`         | The {User} that submitted this Commit for translation. |
 # | `keys`         | All the {Key Keys} found in this Commit.               |
 # | `translations` | The {Translation Translations} found in this Commit.   |
+# | `blobs`        | The {Blob Blobs} found in this Commit.                 |
 #
 # Properties
 # ==========
@@ -68,11 +69,13 @@ require 'fileutils'
 # |:-------------------|:-------------------------------------------------------------------|
 # | `description`      | A user-submitted description of why we are localizing this commit. |
 # | `pull_request_url` | A user-submitted URL to the pull request that is being localized.  |
+# | `author`           | The name of the commit author.                                     |
+# | `author_email`     | The email address of the commit author.                            |
+# | `import_batch_id`  | The ID of the Sidekiq batch of import jobs.                        |

 class Commit < ActiveRecord::Base
   extend RedisMemoize
   include CommitTraverser
-  include SidekiqWorkerTracking

   # @return [true, false] If `true`, does not perform an import after creating
   #   the Commit. Use this to avoid the overhead of making an HTTP request and
@@ -81,16 +84,19 @@ class Commit < ActiveRecord::Base

   belongs_to :project, inverse_of: :commits
   belongs_to :user, inverse_of: :commits
-  has_many :commits_keys, inverse_of: :commit, dependent: :destroy
+  has_many :commits_keys, inverse_of: :commit, dependent: :delete_all
   has_many :keys, through: :commits_keys
   has_many :translations, through: :keys
+  has_many :blobs_commits, inverse_of: :commit, dependent: :delete_all
+  has_many :blobs, through: :blobs_commits

   include HasMetadataColumn
   has_metadata_column(
       description:      {allow_nil: true},
-      author:           {allow_nil: true}, 
+      author:           {allow_nil: true},
       author_email:     {allow_nil: true},
-      pull_request_url: {allow_nil: true}
+      pull_request_url: {allow_nil: true},
+      import_batch_id:  {allow_nil: true}
   )

   include Tire::Model::Search
@@ -146,9 +152,7 @@ class Commit < ActiveRecord::Base

   before_save :set_loaded_at
   before_create :set_author
-  after_commit(on: :create) do |commit|
-    CommitImporter.perform_once(commit.id) unless commit.skip_import
-  end
+  after_commit :initial_import, on: :create
   after_commit :compile_and_cache_or_clear, on: :update
   after_update :update_touchdown_branch
   after_commit :update_stats_at_end_of_loading, on: :update, if: :loading_state_changed?
@@ -333,22 +337,17 @@ class Commit < ActiveRecord::Base
   def import_strings(options={})
     raise CommitNotFoundError, "Commit no longer exists: #{revision}" unless commit!

-    blobs  = project.blobs.includes(:project) # preload blobs for performance
-
-    # add us as one of the workers, to prevent the commit from prematurely going
-    # ready; let's just invent a job ID for us
-    job_id = SecureRandom.uuid
-    add_worker! job_id
+    import_batch.jobs do
+      update_attribute :loading, true
+      blobs = project.blobs.includes(:project) # preload blobs for performance

-    # clear out existing keys so that we can import all new keys
-    keys.clear unless options[:locale]
-    # perform the recursive import
-    traverse(commit!) do |path, blob|
-      import_blob path, blob, options.merge(blobs: blobs)
+      # clear out existing keys so that we can import all new keys
+      commits_keys.delete_all unless options[:locale]
+      # perform the recursive import
+      traverse(commit!) do |path, blob|
+        import_blob path, blob, options.merge(blobs: blobs)
+      end
     end
-
-    # this will also kick of stats recalculation for inline imports
-    remove_worker! job_id
   end

   # Returns a commit object used to interact with Git.
@@ -549,14 +548,32 @@ class Commit < ActiveRecord::Base
   # @private
   def redis_memoize_key() to_param end

-  # @return [true, false] True if there are cached Sidekiq job IDs of
-  #   in-progress BlobImporters that do not actually exist anymore.
+  # @return [Sidekiq::Batch, nil] The batch of Sidekiq workers performing the
+  #   current import, if any.

-  def broken?
-    cached_jids = Shuttle::Redis.smembers("import:#{revision}")
-    return false if cached_jids.empty?
-    actual_jids = self.class.workers.map { |w| w['jid'] }
-    (cached_jids & actual_jids).empty? # none of the cached JIDs actually exist anymore
+  def import_batch
+    if import_batch_id
+      Sidekiq::Batch.new(import_batch_id)
+    else
+      batch             = Sidekiq::Batch.new
+      batch.description = "Import Commit #{id} (#{revision})"
+      batch.on :success, ImportFinisher, commit_id: id
+      update_attribute :import_batch_id, batch.bid
+      batch
+    end
+  rescue Sidekiq::Batch::NoSuchBatch
+    update_attribute :import_batch_id, nil
+    retry
+  end
+
+  # @return [Sidekiq::Batch::Status, nil] Information about the batch of Sidekiq
+  #   workers performing the current import, if any.
+
+  def import_batch_status
+    import_batch_id ? Sidekiq::Batch::Status.new(import_batch_id) : nil
+  rescue Sidekiq::Batch::NoSuchBatch
+    update_attribute :import_batch_id, nil
+    retry
   end

   # Returns whether we should skip a key for this particular commit, given the
@@ -668,7 +685,14 @@ class Commit < ActiveRecord::Base
     imps.each do |importer|
       importer = importer.new(blob_object, path, self)

-      if options[:force]
+      # we can't do a force import on a loading blob -- if we delete all the
+      # blobs_keys while another sidekiq job is doing the import, when that job
+      # finishes the blob will unset loading, even though the former job is still
+      # adding keys to the blob. at this point a third import (with force=false)
+      # might start, see the blob as not loading, and then do a fast import of
+      # the cached keys (even though not all keys have been loaded by the second
+      # import).
+      if options[:force] && !loading?
         blob_object.blobs_keys.delete_all
         blob_object.update_column :loading, true
       end
@@ -681,10 +705,7 @@ class Commit < ActiveRecord::Base
       if options[:inline]
         BlobImporter.new.perform importer.class.ident, project.id, blob.sha, path, id, options[:locale].try!(:rfc5646)
       else
-        shuttle_jid = SecureRandom.uuid
-        blob_object.add_worker! shuttle_jid
-        add_worker! shuttle_jid
-        BlobImporter.perform_once(importer.class.ident, project.id, blob.sha, path, id, options[:locale].try!(:rfc5646), shuttle_jid)
+        BlobImporter.perform_once importer.class.ident, project.id, blob.sha, path, id, options[:locale].try!(:rfc5646)
       end
     end
   end
@@ -700,4 +721,15 @@ class Commit < ActiveRecord::Base
     # calculate readiness and stats.
     CommitStatsRecalculator.new.perform id
   end
+
+  #TODO there's a bug in Rails core that causes this to be run on update as well
+  # as create. sigh.
+  def initial_import
+    return if @_start_transaction_state[:id] # fix bug in Rails core
+    unless skip_import || loading?
+      import_batch.jobs do
+        CommitImporter.perform_once id
+      end
+    end
+  end
 end
diff --git a/app/views/commits/show.slim b/app/views/commits/show.slim
index 597366b..71cc666 100644
--- a/app/views/commits/show.slim
+++ b/app/views/commits/show.slim
@@ -21,10 +21,14 @@
       = "#{@commit.revision[0, 6]}"
       span.separator &nbsp;/&nbsp;
       - if @commit.loading?
-        - current_status = "Loading with #{pluralize(@commit.list_workers.count, 'Job')}"
+        - if @commit.import_batch_status
+          | Currently Loading With #{pluralize_with_delimiter @commit.import_batch_status.pending, 'Job'} Remaining
+        - else
+          | Loading Stalled!
+      - elsif @commit.ready?
+        | Currently Ready
       - else
-        - current_status = (@commit.ready? ? 'Ready' : 'Translating')
-      = "Currently #{current_status}"
+        | Currently Translating
       span.separator &nbsp;/&nbsp;
       - if @commit.completed_at
         = "First Completed #{time_ago_in_words(@commit.completed_at)} ago"
@@ -85,12 +89,6 @@ hr.divider
           | Use this if you need to manually reimport a commit
         .controls
           = button_to 'Reimport', redo_project_commit_path(@project, @commit)
-      - if current_user.admin? and @commit.loading?
-        .control-group
-          label.description-label
-            | Use this to clear workers for hanging commits
-          .controls
-            = button_to 'Clear Workers', clear_project_commit_url(@project, @commit)

       .control-group
         label.description-label
diff --git a/app/views/home/index.slim b/app/views/home/index.slim
index 801bf2b..8a5cb65 100644
--- a/app/views/home/index.slim
+++ b/app/views/home/index.slim
@@ -129,12 +129,17 @@ hr.divider
           - else
             / Progress
             - if commit.loading?
-              td Importing (#{pluralize(commit.list_workers.count, 'Job')})
+              td
+                | Importing
+                - if commit.import_batch_status
+                  |  (#{pluralize_with_delimiter commit.import_batch_status.pending, 'Job'} Remaining)
+                - else
+                  |  (stalled!)
             - elsif commit.ready?
               td Ready for Download
             - else
               - strings_remaining = commit.translations_pending(*@locales) + commit.translations_new(*@locales)
-              td = "Translating #{pluralize(strings_remaining, 'String')}"
+              td = "Translating #{pluralize_with_delimiter strings_remaining, 'String'}"

           / Translation/Monitor Button
           = render partial: 'table_action_button', locals: { commit: commit }
diff --git a/app/views/search/keys.js.coffee.erb b/app/views/search/keys.js.coffee.erb
index 542543e..6360bf7 100644
--- a/app/views/search/keys.js.coffee.erb
+++ b/app/views/search/keys.js.coffee.erb
@@ -112,4 +112,4 @@ $(window).ready ->

   window.onpopstate = ->
     prefillForm()
-    submitSearch()
\ No newline at end of file
+    submitSearch()
diff --git a/app/views/search/translations.js.coffee.erb b/app/views/search/translations.js.coffee.erb
index 1992e63..0df72a3 100644
--- a/app/views/search/translations.js.coffee.erb
+++ b/app/views/search/translations.js.coffee.erb
@@ -69,4 +69,4 @@ $(window).ready ->

   window.onpopstate = ->
     renewSearch() unless firstTime
-    firstTime = false
\ No newline at end of file
+    firstTime = false
diff --git a/app/workers/blob_importer.rb b/app/workers/blob_importer.rb
index f69b94f..7eb74b1 100644
--- a/app/workers/blob_importer.rb
+++ b/app/workers/blob_importer.rb
@@ -29,7 +29,7 @@ class BlobImporter
   #   existing translations from. If `nil`, the base locale is imported as
   #   base translations.

-  def perform(importer, project_id, sha, path, commit_id, rfc5646_locale, shuttle_jid=nil)
+  def perform(importer, project_id, sha, path, commit_id, rfc5646_locale)
     commit  = Commit.find_by_id(commit_id)
     locale  = rfc5646_locale ? Locale.from_rfc5646(rfc5646_locale) : nil
     project = Project.find(project_id)
@@ -38,7 +38,9 @@ class BlobImporter
     if blob.blob!.nil?
       # for whatever reason sometimes the blob is not accessible; try again in
       # 5 minutes
-      BlobImporter.perform_in(5.minutes, importer, project_id, sha, path, commit_id, rfc5646_locale, shuttle_jid)
+      commit.import_batch.jobs do
+        BlobImporter.perform_in 5.minutes, importer, project_id, sha, path, commit_id, rfc5646_locale
+      end
       return
     end

@@ -49,9 +51,6 @@ class BlobImporter
                         commit: commit,
                         locale: locale,
                         inline: jid.nil?
-
-    blob.remove_worker! shuttle_jid
-    commit.remove_worker! shuttle_jid
   end

   include SidekiqLocking
diff --git a/app/workers/key_creator.rb b/app/workers/key_creator.rb
index edc64ce..1f92dfd 100644
--- a/app/workers/key_creator.rb
+++ b/app/workers/key_creator.rb
@@ -45,12 +45,8 @@ class KeyCreator
     key_objects.map(&:id).uniq.each { |k| @blob.blobs_keys.where(key_id: k).find_or_create! }

     if @commit
-      key_objects.reject! { |key| skip_key?(key) }
       self.class.update_key_associations key_objects, @commit
     end
-
-    @blob.remove_worker! shuttle_jid
-    @commit.remove_worker! shuttle_jid if @commit
   end

   # Given a set of keys, bulk-updates their commits-keys associations and
@@ -60,6 +56,7 @@ class KeyCreator
   # @param [Commit] commit A Commit these keys are associated with.

   def self.update_key_associations(keys, commit)
+    keys.reject! { |key| skip_key?(key, commit) }
     keys.map(&:id).uniq.each { |k| commit.commits_keys.where(key_id: k).find_or_create! }

     # key.commits has been changed, need to update associated ES fields
@@ -112,16 +109,15 @@ class KeyCreator

   # Determines if we should skip this key using both the normal key exclusions
   # and the .shuttle.yml key exclusions
-  def skip_key?(key)
-    skip_key_due_to_project_settings?(key) || skip_key_due_to_branch_settings?(key)
+  def self.skip_key?(key, commit)
+    skip_key_due_to_project_settings?(key, commit.project) || skip_key_due_to_branch_settings?(key, commit)
   end

-  def skip_key_due_to_project_settings?(key)
-    @blob.project.skip_key?(key.key, @blob.project.base_locale)
+  def self.skip_key_due_to_project_settings?(key, project)
+    project.skip_key?(key.key, project.base_locale)
   end

-  def skip_key_due_to_branch_settings?(key)
-    return false unless @commit
-    @commit.skip_key?(key.key)
+  def self.skip_key_due_to_branch_settings?(key, commit)
+    commit.skip_key?(key.key)
   end
 end
diff --git a/config/routes.rb b/config/routes.rb
index da45a48..3113e44 100644
--- a/config/routes.rb
+++ b/config/routes.rb
@@ -40,7 +40,7 @@ Shuttle::Application.routes.draw do
   resources :projects do
     resources :commits, only: [:show, :create, :update, :destroy] do
       member do
-        post :import, :sync, :redo, :clear, :recalculate
+        post :import, :sync, :redo, :recalculate
         get :manifest, :localize
       end

diff --git a/db/migrate/20140320053508_create_blobs_commits.rb b/db/migrate/20140320053508_create_blobs_commits.rb
new file mode 100644
index 0000000..ab6cca1
--- /dev/null
+++ b/db/migrate/20140320053508_create_blobs_commits.rb
@@ -0,0 +1,31 @@
+# Copyright 2013 Square Inc.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+class CreateBlobsCommits < ActiveRecord::Migration
+  def up
+    execute <<-SQL
+      CREATE TABLE blobs_commits(
+        project_id INTEGER NOT NULL,
+        sha_raw BYTEA NOT NULL,
+        commit_id INTEGER NOT NULL REFERENCES commits(id) ON DELETE CASCADE,
+        FOREIGN KEY (project_id, sha_raw) REFERENCES blobs(project_id, sha_raw) ON DELETE CASCADE,
+        PRIMARY KEY (project_id, sha_raw, commit_id)
+      )
+    SQL
+  end
+
+  def down
+    drop_table :blobs_commits
+  end
+end
diff --git a/db/structure.sql b/db/structure.sql
index bef994d..075b757 100644
--- a/db/structure.sql
+++ b/db/structure.sql
@@ -41,6 +41,17 @@ CREATE TABLE blobs (


 --
+-- Name: blobs_commits; Type: TABLE; Schema: public; Owner: -; Tablespace: 
+--
+
+CREATE TABLE blobs_commits (
+    project_id integer NOT NULL,
+    sha_raw bytea NOT NULL,
+    commit_id integer NOT NULL
+);
+
+
+--
 -- Name: blobs_keys; Type: TABLE; Schema: public; Owner: -; Tablespace: 
 --

@@ -551,6 +562,14 @@ ALTER TABLE ONLY users ALTER COLUMN id SET DEFAULT nextval('users_id_seq'::regcl


 --
+-- Name: blobs_commits_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
+--
+
+ALTER TABLE ONLY blobs_commits
+    ADD CONSTRAINT blobs_commits_pkey PRIMARY KEY (project_id, sha_raw, commit_id);
+
+
+--
 -- Name: blobs_keys_pkey; Type: CONSTRAINT; Schema: public; Owner: -; Tablespace: 
 --

@@ -797,6 +816,22 @@ CREATE UNIQUE INDEX users_unlock_token ON users USING btree (unlock_token);


 --
+-- Name: blobs_commits_commit_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: -
+--
+
+ALTER TABLE ONLY blobs_commits
+    ADD CONSTRAINT blobs_commits_commit_id_fkey FOREIGN KEY (commit_id) REFERENCES commits(id) ON DELETE CASCADE;
+
+
+--
+-- Name: blobs_commits_project_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: -
+--
+
+ALTER TABLE ONLY blobs_commits
+    ADD CONSTRAINT blobs_commits_project_id_fkey FOREIGN KEY (project_id, sha_raw) REFERENCES blobs(project_id, sha_raw) ON DELETE CASCADE;
+
+
+--
 -- Name: blobs_keys_key_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: -
 --

@@ -999,3 +1034,5 @@ INSERT INTO schema_migrations (version) VALUES ('20140228025058');
 INSERT INTO schema_migrations (version) VALUES ('20140306064700');

 INSERT INTO schema_migrations (version) VALUES ('20140311011156');
+
+INSERT INTO schema_migrations (version) VALUES ('20140320053508');
diff --git a/lib/import_finisher.rb b/lib/import_finisher.rb
new file mode 100644
index 0000000..e4f1ebb
--- /dev/null
+++ b/lib/import_finisher.rb
@@ -0,0 +1,16 @@
+# Contains hooks run by Sidekiq upon completion of an import batch.
+
+class ImportFinisher
+
+  # Run by Sidekiq after an import batch finishes successfully. Unsets the
+  # {Commit}'s `loading` flag (thus running post-import hooks), and unsets the
+  # loading flag on all associated {Blob Blobs}.
+
+  def on_success(_status, options)
+    commit = Commit.find(options['commit_id'])
+    commit.update_attributes loading: false, import_batch_id: nil
+    # commit.blobs.update_all loading: false
+    blob_shas = commit.blobs_commits.pluck(:sha_raw)
+    Blob.where(project_id: commit.project_id, sha_raw: blob_shas).update_all loading: false
+  end
+end
diff --git a/lib/importer/base.rb b/lib/importer/base.rb
index 4a1d2c8..b2315d9 100644
--- a/lib/importer/base.rb
+++ b/lib/importer/base.rb
@@ -95,6 +95,10 @@ module Importer
       @blob   = blob
       @commit = commit
       @file   = File.new(path, nil, nil)
+
+      if @commit
+        blob.blobs_commits.where(commit_id: @commit.id).find_or_create!
+      end
     end

     # Scans the Blob for localizable strings, and creates or updates
@@ -283,6 +287,8 @@ module Importer
     end

     def import_by_parsing_blob
+      @blob.update_attribute :loading, true
+
       load_contents

       @keys = Array.new
@@ -293,14 +299,20 @@ module Importer
       end

       # then spawn jobs to create those keys
-      @keys.in_groups_of(100, false) do |keys|
-        if inline
+      if inline
+        @keys.in_groups_of(100, false) do |keys|
           KeyCreator.new.perform @blob.project_id, @blob.sha, @commit.try!(:id), self.class.ident, keys
-        else
-          shuttle_jid = SecureRandom.uuid
-          @blob.add_worker! shuttle_jid
-          @commit.add_worker!(shuttle_jid) if @commit
-          KeyCreator.perform_async(@blob.project_id, @blob.sha, @commit.try!(:id), self.class.ident, keys, shuttle_jid)
+        end
+      elsif @commit
+        bulk_args = @keys.in_groups_of(100, false).map do |keys|
+          [@blob.project_id, @blob.sha, @commit.try!(:id), self.class.ident, keys]
+        end
+        @commit.import_batch.jobs do
+          Sidekiq::Client.push_bulk 'class' => KeyCreator, 'args' => bulk_args
+        end
+      else
+        @keys.in_groups_of(100, false) do |keys|
+          KeyCreator.perform_async @blob.project_id, @blob.sha, @commit.try!(:id), self.class.ident, keys
         end
       end
     end
@@ -308,7 +320,7 @@ module Importer
     # Used when this blob was imported as part of an earlier commit; just
     # associates the cached list of keys for that blob with the new commit
     def import_by_using_cached_keys
-      KeyCreator.update_key_associations @blob.keys, @commit
+      KeyCreator.update_key_associations @blob.keys.to_a, @commit
     end

     # array indexes are stored in brackets
diff --git a/lib/sidekiq_locking.rb b/lib/sidekiq_locking.rb
index b679f4f..5aab039 100644
--- a/lib/sidekiq_locking.rb
+++ b/lib/sidekiq_locking.rb
@@ -49,7 +49,7 @@ module SidekiqLocking
     end

     def mutex(*args)
-      Redis::Mutex.new(lock_name(*args), :expire => 1.hour)
+      Redis::Mutex.new(lock_name(*args), expire: 1.hour)
     end
   end
 end
diff --git a/lib/sidekiq_worker_tracking.rb b/lib/sidekiq_worker_tracking.rb
deleted file mode 100644
index 855e1a2..0000000
--- a/lib/sidekiq_worker_tracking.rb
+++ /dev/null
@@ -1,97 +0,0 @@
-# Copyright 2014 Square Inc.
-#
-#    Licensed under the Apache License, Version 2.0 (the "License");
-#    you may not use this file except in compliance with the License.
-#    You may obtain a copy of the License at
-#
-#        http://www.apache.org/licenses/LICENSE-2.0
-#
-#    Unless required by applicable law or agreed to in writing, software
-#    distributed under the License is distributed on an "AS IS" BASIS,
-#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#    See the License for the specific language governing permissions and
-#    limitations under the License.
-
-# Adds the ability to track Sidekiq workers performing an oepration on an object
-# and update the value of a `loading` column when all workers have completed.
-#
-# To use this module, your model must have an attribute named `loading`. When
-# you add a worker that performs an operation on your model, you should pass its
-# JID to the {#add_worker!} method. The model's `loading` column will be set to
-# `false`, and will remain `false` until all such workers have completed.
-#
-# @example
-#   class MyModel
-#     include SidekiqWorkerTracking
-#     attr_accessor :loading
-#
-#     def perform_operation
-#       10.times do
-#         add_worker! OperationPerformer.perform_async(...)
-#       end
-#     end
-#
-#     def all_operations_completed?
-#       !loading
-#     end
-#   end
-
-module SidekiqWorkerTracking
-  # Adds a worker to the loading list. This object, if not already loading, will
-  # be marked as loading until this and all other added workers call
-  # {#remove_worker!}.
-  #
-  # @param [String] jid A unique identifier for this worker.
-
-  def add_worker!(jid)
-    self.loading = true
-    save!
-    Shuttle::Redis.sadd worker_set_key, jid
-  end
-
-  # Removes a worker from the loading list. This object will not be marked as
-  # loading if this was the last worker. Also recalculates Commit statistics if
-  # this was the last worker.
-  #
-  # @param [String] jid A unique identifier for this worker.
-  # @see #add_worker!
-
-  def remove_worker!(jid)
-    if jid.nil?
-      return
-    end
-
-    unless Shuttle::Redis.srem(worker_set_key, jid)
-      Squash::Ruby.record "Failed to remove worker", object: self, jid: jid, unimportant: true
-    end
-
-    loading = (Shuttle::Redis.scard(worker_set_key) > 0)
-
-    self.loading = loading
-    save!
-  end
-
-  # Returns all workers from the loading list
-
-  def list_workers
-    Shuttle::Redis.smembers worker_set_key
-  end
-
-  # Removes all workers from the loading list, marks the Commit as not loading,
-  # and recalculates Commit statistics if the Commit was previously loading.
-  # This method should be used to fix "stuck" Commits.
-
-  def clear_workers!
-    Shuttle::Redis.del worker_set_key
-    if loading?
-      self.loading = false
-      save!
-    end
-  end
-
-  private
-
-  def worker_set_key
-    "loading:#{self.class.to_s}:#{self.id}"
-  end
-end
diff --git a/lib/tasks/maintenance.rake b/lib/tasks/maintenance.rake
index 9e36ba9..e5a3ad1 100644
--- a/lib/tasks/maintenance.rake
+++ b/lib/tasks/maintenance.rake
@@ -13,11 +13,6 @@
 #    limitations under the License.

 namespace :maintenance do
-  desc "Locate hung commits and clear their worker queue"
-  task fix_hung_commits: :environment do
-    Commit.where(loading: true).select(&:broken?).each(&:clear_workers!)
-  end
-
   desc "Locates lockfiles that no longer refer to active processes and clears them"
   task clear_stale_lockfiles: :environment do
     workers      = Shuttle::Redis.smembers('workers').select { |w| w.start_with? Socket.gethostname }
diff --git a/spec/controllers/search_controller_spec.rb b/spec/controllers/search_controller_spec.rb
index a5e39c2..271b41b 100644
--- a/spec/controllers/search_controller_spec.rb
+++ b/spec/controllers/search_controller_spec.rb
@@ -77,10 +77,11 @@ describe SearchController do
       expect(response.status).to eql(200)
       results = JSON.parse(response.body)
       expect(results.size).to eql(2)
-      expect(results.first['copy']).to eql('foo term1 bar')
-      expect(results.first['locale']['rfc5646']).to eql('fr')
+      results.sort_by! { |r| r['locale']['rfc5646'] }
       expect(results.last['copy']).to eql('foo term1 bar')
-      expect(results.last['locale']['rfc5646']).to eql('en')
+      expect(results.last['locale']['rfc5646']).to eql('fr')
+      expect(results.first['copy']).to eql('foo term1 bar')
+      expect(results.first['locale']['rfc5646']).to eql('en')
     end

     it "should respond with a 422 if the locale is unknown" do
diff --git a/spec/factories/blobs_commits.rb b/spec/factories/blobs_commits.rb
new file mode 100644
index 0000000..12d6e63
--- /dev/null
+++ b/spec/factories/blobs_commits.rb
@@ -0,0 +1,6 @@
+FactoryGirl.define do
+  factory :blobs_commit do
+    association :blob
+    association :commit
+  end
+end
diff --git a/spec/factories/blobs_keys.rb b/spec/factories/blobs_keys.rb
index 023107a..bd8b8c9 100644
--- a/spec/factories/blobs_keys.rb
+++ b/spec/factories/blobs_keys.rb
@@ -14,7 +14,7 @@

 FactoryGirl.define do
   factory :blobs_key do
-    association :blobs
-    association :keys
+    association :blob
+    association :key
   end
 end
diff --git a/spec/models/blobs_commit_spec.rb b/spec/models/blobs_commit_spec.rb
new file mode 100644
index 0000000..ed0762d
--- /dev/null
+++ b/spec/models/blobs_commit_spec.rb
@@ -0,0 +1,4 @@
+require 'spec_helper'
+
+describe BlobsCommit do
+end
diff --git a/spec/models/commit_spec.rb b/spec/models/commit_spec.rb
index 6832813..c4f19df 100644
--- a/spec/models/commit_spec.rb
+++ b/spec/models/commit_spec.rb
@@ -550,6 +550,26 @@ describe Commit do
       @project.commit! 'HEAD'
       expect(@project.keys.map(&:importer).uniq.sort).to eql(Importer::Base.implementations.map(&:ident).sort - %w(yaml))
     end
+
+    it "should remove appropriate keys when reimporting after changed settings" do
+      commit = @project.commit!('HEAD')
+      expect(commit.keys.map(&:original_key)).to include('root')
+
+      @project.update_attribute :key_exclusions, %w(roo*)
+      commit.import_strings
+      expect(commit.keys(true).map(&:original_key)).not_to include('root')
+    end
+
+    it "should only associate relevant keys with a new commit when cached blob importing is being use3d" do
+      @project.update_attribute :key_exclusions, %w(skip_me)
+      commit      = @project.commit!('HEAD')
+      blob        = commit.blobs.first
+      red_herring = FactoryGirl.create(:key, key: 'skip_me')
+      FactoryGirl.create :blobs_key, key: red_herring, blob: blob
+
+      commit.import_strings
+      expect(commit.keys(true)).not_to include(red_herring)
+    end
   end

   describe "#all_translations_entered_for_locale?" do

Can I just have a try without buying sidekiq pro #108

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions