From what the timeline says, Doflamingo took Dressrosa for his own ten years before the present and that Caesar started his criminal career about six years after. So when did the Dwarves get shanghaied? If there wasn't always a SMILE Factory, what would Joker be profiting from aside from the usual Underworld stuff? From what I understand, he didn't thought of SMILES until Caesar came into the picture.
See what I'm getting at? I'm wondering if there's a plot hole, or maybe I'm just overthinking.