Man this is tough, I had to put a ton of thought into this. Mostly because in the last Decade and a half I would say that RPGs have strived not to be overly complex. One of the last remaining kings of complex games that are deeply rewarding would be Shadowrun.
But it is not my pick actually. I will be going with a game that has a MASSIVE library. It is deeply complex and very rewarding. Because of it, if you spend enough time I don’t think there is anything that you can not create with this system. G.U.R.P.S.
It is an oldie but a goodie and I imagine I can find several gamers that would stand up and cheer for this selection. GURPS for years and years has basically been a game with cult status. Everyone knows it and few have played it. Those that have played it swear by it.
I would love to track down some of my old GURPS Gm’s and get into a fresh game of GURPS after all these years. Its THICK its COMPLEX but with a Good GM its a thing of beauty.