Once you have created installer using installshield and able to build through GUI then you can build the same in jenkins like below using windows batch command-
rd /Q /S “Your Project/staging”
mkdir “Your Project/staging”
xcopy “Path to your project components\*” “Your Project\staging” /e /f /i
“E:\Program Files\Installshield2013\System\IsCmdBld.exe” -p “%WORKSPACE%\Your Project\Your Project.ism” -y %BUILDNO%
where, you can provide BUILDNO pre your project needs.
Basically, what does above program do is remove older staging folder, create empty staging folder, copy contents there and then build installer from there. You need to link your components to your staging folder in install shield project file.