From the OED. The predominant use of "Luddite" has been disparaging for a long time.
You can debate whether it should be or not, but it's not reasonable to criticize conventional use.