Discussion:
[WiX-users] How to debug CustomAction DLL
little.forest
2010-08-24 21:11:57 UTC
Permalink
Dear Wix Experts:

We have a CustomAction DLL written in C. How to debug it?

There are a few posts about it:
Post1:
http://www.davidmoore.info/2010/06/28/how-to-debug-a-windows-installer-custom-action/

Post2: http://blogs.msdn.com/b/astebner/archive/2005/06/17/430320.aspx
Post3: http://msdn.microsoft.com/en-us/library/aa368264(VS.85).aspx

I tried it in Windows 7. I did set MsiBreak, but that famous "message box" never
showed up. So I tried it in XP, the "message box" showed up. It says "To debug
your custom action, attach your debugger to process 5632(0x1600) and press OK".
I opened Windbg.

Post1 says "attach the process", Post2 says "Open Executable...". I tried both,
neither works for me. In both cases, I got "*** ERROR: Symbol file could not be
found. Defaulted to export symbols for C:\WINDOWS\system32\ntdll.dll -
ntdll!DbgBreakPoint: 7c90120e cc int 3" in Windbg.

Questions:
1. What's the problem here?
2. Should I attach process? Or should I "Open Executable..."?
3. If I should attach process, why it doesn't work?
4. If I should "Open Executable...", what the "File name" and "Arguments" fields
should be? For example, the "File name" should be "C:\Temp\MyApp.msi" or
"C:\Windows\system32\msiexec.exe"? How about arguments? Should it be "/i
C:\Temp\MyApp.msi"?
5. Why it doesn't work in Windows 7?

If you even debug DLL CustomAction, pls let me know.

Thanks!
Tony Juricic
2010-08-24 22:00:47 UTC
Permalink
Good ole' MessageBox is the most reliable way to debug C/C++ CAs that I found. You may be able to set the breakpoint after the MessageBox and then attach to the running msiexec process.

-----Original Message-----
From: little.forest [mailto:***@ymail.com]
Sent: Tuesday, August 24, 2010 5:12 PM
To: wix-***@lists.sourceforge.net
Subject: [WiX-users] How to debug CustomAction DLL

Dear Wix Experts:

We have a CustomAction DLL written in C. How to debug it?

There are a few posts about it:
Post1:
http://www.davidmoore.info/2010/06/28/how-to-debug-a-windows-installer-custom-action/

Post2: http://blogs.msdn.com/b/astebner/archive/2005/06/17/430320.aspx
Post3: http://msdn.microsoft.com/en-us/library/aa368264(VS.85).aspx

I tried it in Windows 7. I did set MsiBreak, but that famous "message box" never
showed up. So I tried it in XP, the "message box" showed up. It says "To debug
your custom action, attach your debugger to process 5632(0x1600) and press OK".
I opened Windbg.

Post1 says "attach the process", Post2 says "Open Executable...". I tried both,
neither works for me. In both cases, I got "*** ERROR: Symbol file could not be
found. Defaulted to export symbols for C:\WINDOWS\system32\ntdll.dll -
ntdll!DbgBreakPoint: 7c90120e cc int 3" in Windbg.

Questions:
1. What's the problem here?
2. Should I attach process? Or should I "Open Executable..."?
3. If I should attach process, why it doesn't work?
4. If I should "Open Executable...", what the "File name" and "Arguments" fields
should be? For example, the "File name" should be "C:\Temp\MyApp.msi" or
"C:\Windows\system32\msiexec.exe"? How about arguments? Should it be "/i
C:\Temp\MyApp.msi"?
5. Why it doesn't work in Windows 7?

If you even debug DLL CustomAction, pls let me know.

Thanks!



TradeStation Group, Inc. is a publicly-traded holding company (NASDAQ GS: TRAD) of three operating subsidiaries, TradeStation Securities, Inc. (Member NYSE, FINRA, SIPC and NFA), TradeStation Technologies, Inc., a trading software and subscription company, and TradeStation Europe Limited, a United Kingdom, FSA-authorized introducing brokerage firm. None of these companies provides trading or investment advice, recommendations or endorsements of any kind. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
Christopher Painter
2010-08-25 00:12:44 UTC
Permalink
I think I recall you sometimes have to reboot and/or restart the Msiexec service
after setting MsiBreak because the Service Control Manager doesn't get the env
update.  I haven't used it in a long time so I could be wrong.  on the C#/DTF
side I tend to use MessageBox during intitial development and almost never need
to debug a custom action later in it's life cycle.
 



----- Original Message ----
From: Tony Juricic <***@TradeStation.com>
To: General discussion for Windows Installer XML toolset.
<wix-***@lists.sourceforge.net>
Sent: Tue, August 24, 2010 5:00:47 PM
Subject: Re: [WiX-users] How to debug CustomAction DLL

Good ole' MessageBox is the most reliable way to debug C/C++ CAs that I found.
You may be able to set the breakpoint after the MessageBox and then attach to
the running msiexec process.


-----Original Message-----
From: little.forest [mailto:***@ymail.com]
Sent: Tuesday, August 24, 2010 5:12 PM
To: wix-***@lists.sourceforge.net
Subject: [WiX-users] How to debug CustomAction DLL

Dear Wix Experts:

We have a CustomAction DLL written in C. How to debug it?

There are a few posts about it:
Post1:
http://www.davidmoore.info/2010/06/28/how-to-debug-a-windows-installer-custom-action/


Post2: http://blogs.msdn.com/b/astebner/archive/2005/06/17/430320.aspx
Post3: http://msdn.microsoft.com/en-us/library/aa368264(VS.85).aspx

I tried it in Windows 7. I did set MsiBreak, but that famous "message box" never

showed up. So I tried it in XP, the "message box" showed up. It says "To debug
your custom action, attach your debugger to process 5632(0x1600) and press OK".
I opened Windbg.

Post1 says "attach the process", Post2 says "Open Executable...". I tried both,
neither works for me. In both cases, I got "*** ERROR: Symbol file could not be
found.  Defaulted to export symbols for C:\WINDOWS\system32\ntdll.dll -
ntdll!DbgBreakPoint: 7c90120e cc              int    3" in Windbg.

Questions:
1. What's the problem here?
2. Should I attach process? Or should I "Open Executable..."?
3. If I should attach process, why it doesn't work?
4. If I should "Open Executable...", what the "File name" and "Arguments" fields

should be? For example, the "File name" should be "C:\Temp\MyApp.msi" or
"C:\Windows\system32\msiexec.exe"? How about arguments? Should it be "/i
C:\Temp\MyApp.msi"?
5. Why it doesn't work in Windows 7?

If you even debug DLL CustomAction, pls let me know.

Thanks!



TradeStation Group, Inc. is a publicly-traded holding company (NASDAQ GS: TRAD)
of three operating subsidiaries, TradeStation Securities, Inc. (Member NYSE,
FINRA, SIPC and NFA), TradeStation Technologies, Inc., a trading software and
subscription company, and TradeStation Europe Limited, a United Kingdom,
FSA-authorized introducing brokerage firm. None of these companies provides
trading or investment advice, recommendations or endorsements of any kind. The
information transmitted is intended only for the person or entity to which it is
addressed and may contain confidential and/or privileged material. Any review,
retransmission, dissemination or other use of, or taking of any action in
reliance upon, this information by persons or entities other than the intended
recipient is prohibited. If you received this in error, please contact the
sender and delete the material from any computer.
Wilson, Phil
2010-08-25 16:43:47 UTC
Permalink
You may be able to use the MSI logging if you want to trace data into the MSI log file. Another cheap trick I've used is to set interestingly named properties to whatever I want to see the values of. The setting of these properties and values is logged.

Phil Wilson

-----Original Message-----
From: Christopher Painter [mailto:***@deploymentengineering.com]
Sent: Tuesday, August 24, 2010 5:13 PM
To: General discussion for Windows Installer XML toolset.
Subject: Re: [WiX-users] How to debug CustomAction DLL

I think I recall you sometimes have to reboot and/or restart the Msiexec service
after setting MsiBreak because the Service Control Manager doesn't get the env
update.  I haven't used it in a long time so I could be wrong.  on the C#/DTF
side I tend to use MessageBox during intitial development and almost never need
to debug a custom action later in it's life cycle.
 



----- Original Message ----
From: Tony Juricic <***@TradeStation.com>
To: General discussion for Windows Installer XML toolset.
<wix-***@lists.sourceforge.net>
Sent: Tue, August 24, 2010 5:00:47 PM
Subject: Re: [WiX-users] How to debug CustomAction DLL

Good ole' MessageBox is the most reliable way to debug C/C++ CAs that I found.
You may be able to set the breakpoint after the MessageBox and then attach to
the running msiexec process.


-----Original Message-----
From: little.forest [mailto:***@ymail.com]
Sent: Tuesday, August 24, 2010 5:12 PM
To: wix-***@lists.sourceforge.net
Subject: [WiX-users] How to debug CustomAction DLL

Dear Wix Experts:

We have a CustomAction DLL written in C. How to debug it?

There are a few posts about it:
Post1:
http://www.davidmoore.info/2010/06/28/how-to-debug-a-windows-installer-custom-action/


Post2: http://blogs.msdn.com/b/astebner/archive/2005/06/17/430320.aspx
Post3: http://msdn.microsoft.com/en-us/library/aa368264(VS.85).aspx

I tried it in Windows 7. I did set MsiBreak, but that famous "message box" never

showed up. So I tried it in XP, the "message box" showed up. It says "To debug
your custom action, attach your debugger to process 5632(0x1600) and press OK".
I opened Windbg.

Post1 says "attach the process", Post2 says "Open Executable...". I tried both,
neither works for me. In both cases, I got "*** ERROR: Symbol file could not be
found.  Defaulted to export symbols for C:\WINDOWS\system32\ntdll.dll -
ntdll!DbgBreakPoint: 7c90120e cc              int    3" in Windbg.

Questions:
1. What's the problem here?
2. Should I attach process? Or should I "Open Executable..."?
3. If I should attach process, why it doesn't work?
4. If I should "Open Executable...", what the "File name" and "Arguments" fields

should be? For example, the "File name" should be "C:\Temp\MyApp.msi" or
"C:\Windows\system32\msiexec.exe"? How about arguments? Should it be "/i
C:\Temp\MyApp.msi"?
5. Why it doesn't work in Windows 7?

If you even debug DLL CustomAction, pls let me know.

Thanks!



TradeStation Group, Inc. is a publicly-traded holding company (NASDAQ GS: TRAD)
of three operating subsidiaries, TradeStation Securities, Inc. (Member NYSE,
FINRA, SIPC and NFA), TradeStation Technologies, Inc., a trading software and
subscription company, and TradeStation Europe Limited, a United Kingdom,
FSA-authorized introducing brokerage firm. None of these companies provides
trading or investment advice, recommendations or endorsements of any kind. The
information transmitted is intended only for the person or entity to which it is
addressed and may contain confidential and/or privileged material. Any review,
retransmission, dissemination or other use of, or taking of any action in
reliance upon, this information by persons or entities other than the intended
recipient is prohibited. If you received this in error, please contact the
sender and delete the material from any computer.

------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users
worldwide. Take advantage of special opportunities to increase revenue and
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d
_______________________________________________
WiX-users mailing list
WiX-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wix-users





------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users
worldwide. Take advantage of special opportunities to increase revenue and
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d
_______________________________________________
WiX-users mailing list
WiX-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/wix-users


*** Confidentiality Notice: This e-mail, including any associated or attached files, is intended solely for the individual or entity to which it is addressed. This e-mail is confidential and may well also be legally privileged. If you have received it in error, you are on notice of its status. Please notify the sender immediately by reply e-mail and then delete this message from your system. Please do not copy it or use it for any purposes, or disclose its contents to any other person. This email comes from a division of the Invensys Group, owned by Invensys plc, which is a company registered in England and Wales with its registered office at 3rd Floor, 40 Grosvenor Place, London, SW1X 7AW (Registered number 166023). For a list of European legal entities within the Invensys Group, please go to http://www.invensys.com/legal/default.asp?top_nav_id=77&nav_id=80&prev_id=77.

You may contact Invensys plc on +44 (0)20 3155 1200 or e-mail ***@invensys.com. This e-mail and any attachments thereto may be subject to the terms of any agreements between Invensys (and/or its subsidiaries and affiliates) and the recipient (and/or its subsidiaries and affiliates).
dB.
2010-08-25 11:18:33 UTC
Permalink
If you're developing a custom action (or many), you might save yourself a lot of time in the future by investing a bit upfront into unit testing your CA. This would allow you to invoke it from CPPUNIT, assert conditions, and obvious debug it without message boxes.

There's a post about the approach here - http://code.dblock.org/ShowPost.aspx?id=7 - and it's fully implemented in AppSecInc. community MSI extensions - http://msiext.codeplex.com.

I haven't had to stick a message box in a CA for a while to debug it "live" :)

dB. @ dblock.org
Moscow|Geneva|Seattle|New York


-----Original Message-----
From: little.forest [mailto:***@ymail.com]
Sent: Tuesday, August 24, 2010 5:12 PM
To: wix-***@lists.sourceforge.net
Subject: [WiX-users] How to debug CustomAction DLL

Dear Wix Experts:

We have a CustomAction DLL written in C. How to debug it?

There are a few posts about it:
Post1:
http://www.davidmoore.info/2010/06/28/how-to-debug-a-windows-installer-custom-action/

Post2: http://blogs.msdn.com/b/astebner/archive/2005/06/17/430320.aspx
Post3: http://msdn.microsoft.com/en-us/library/aa368264(VS.85).aspx

I tried it in Windows 7. I did set MsiBreak, but that famous "message box" never
showed up. So I tried it in XP, the "message box" showed up. It says "To debug
your custom action, attach your debugger to process 5632(0x1600) and press OK".
I opened Windbg.

Post1 says "attach the process", Post2 says "Open Executable...". I tried both,
neither works for me. In both cases, I got "*** ERROR: Symbol file could not be
found. Defaulted to export symbols for C:\WINDOWS\system32\ntdll.dll -
ntdll!DbgBreakPoint: 7c90120e cc int 3" in Windbg.

Questions:
1. What's the problem here?
2. Should I attach process? Or should I "Open Executable..."?
3. If I should attach process, why it doesn't work?
4. If I should "Open Executable...", what the "File name" and "Arguments" fields
should be? For example, the "File name" should be "C:\Temp\MyApp.msi" or
"C:\Windows\system32\msiexec.exe"? How about arguments? Should it be "/i
C:\Temp\MyApp.msi"?
5. Why it doesn't work in Windows 7?

If you even debug DLL CustomAction, pls let me know.

Thanks!
jhennessey
2010-08-25 18:17:16 UTC
Permalink
I set the environment variable MsiBreak for native code (
http://msdn.microsoft.com/en-us/library/aa368264%28VS.85%29.aspx
http://msdn.microsoft.com/en-us/library/aa368264%28VS.85%29.aspx ) and
MMsiBreak for managed code (
http://blog.torresdal.net/2008/10/26/WiXAndDTFDebugAManagedCustomActionAndHowToGenerateAnMSILog.aspx
http://blog.torresdal.net/2008/10/26/WiXAndDTFDebugAManagedCustomActionAndHowToGenerateAnMSILog.aspx
) to the name of the custom action to debug. This way you don't have to mess
with adding message boxes.

As was already mentioned, sometimes setting environment variables doesn't
work until you reboot so what I do is launch a command prompt, set the
variable (using the SET command), then launch the install from the command
prompt (this always works for me).
--
View this message in context: http://windows-installer-xml-wix-toolset.687559.n2.nabble.com/How-to-debug-CustomAction-DLL-tp5458781p5462437.html
Sent from the wix-users mailing list archive at Nabble.com.
gapearce
2010-08-25 21:00:28 UTC
Permalink
I found it pretty easy and straightforward using VS2008 to attach to a setup
program awaiting user input from a message box. It was very easy...

This MSND article touches on it -
http://msdn.microsoft.com/en-us/library/aa368264%28VS.85%29.aspx -

just attach to the process that has your messagebox title in it.

add a breakpoint in VS2008 on the line of code in the dll after the
messagebox.
Of course the code has to match the dll...

Respond to the button in the messagebox. It will close and stop on the line
in the debugger.

You do have to remember to remove / disable the message boxes as stated
above...when you ship, or when they become so annoying that you can't take
it any more.
--
View this message in context: http://windows-installer-xml-wix-toolset.687559.n2.nabble.com/How-to-debug-CustomAction-DLL-tp5458781p5463104.html
Sent from the wix-users mailing list archive at Nabble.com.
Rob Mensching
2010-09-07 03:37:54 UTC
Permalink
I like to use the AssertSz(FALSE, "debug XxxCA here"); from dutil.h. The
resulting dialog gives lots of information to make it easy to attach.
Post by gapearce
I found it pretty easy and straightforward using VS2008 to attach to a setup
program awaiting user input from a message box. It was very easy...
This MSND article touches on it -
http://msdn.microsoft.com/en-us/library/aa368264%28VS.85%29.aspx -
just attach to the process that has your messagebox title in it.
add a breakpoint in VS2008 on the line of code in the dll after the
messagebox.
Of course the code has to match the dll...
Respond to the button in the messagebox. It will close and stop on the line
in the debugger.
You do have to remember to remove / disable the message boxes as stated
above...when you ship, or when they become so annoying that you can't take
it any more.
--
http://windows-installer-xml-wix-toolset.687559.n2.nabble.com/How-to-debug-CustomAction-DLL-tp5458781p5463104.html
Sent from the wix-users mailing list archive at Nabble.com.
------------------------------------------------------------------------------
Sell apps to millions through the Intel(R) Atom(Tm) Developer Program
Be part of this innovative community and reach millions of netbook users
worldwide. Take advantage of special opportunities to increase revenue and
speed time-to-market. Join now, and jumpstart your future.
http://p.sf.net/sfu/intel-atom-d2d
_______________________________________________
WiX-users mailing list
https://lists.sourceforge.net/lists/listinfo/wix-users
--
virtually, Rob Mensching - http://RobMensching.com LLC
Loading...