Echidina
Sophie and Jon
Michelle, Sylvia, Sophie and Maria all decked out
Sue and Dean
the west coast of South New Zealand
with ice sculptures everywhere
Wendy and Anh (c/o Wendy Rourke)
Andrew brooding

 

February 2007
M T W T F S S
« Jan   Apr »
 1234
567891011
12131415161718
19202122232425
262728293031  

vmware remote console won’t connect!

VMware logohmmm, beginning to think that the biggest problem that I've had with vmware is that I couldn't find a logo for the post ... spent a good 5 minutes looking for one!
I've been asked at work to setup a number of new vmware machines on a virtualised server in a completely different location (not easily geographically reached). This would be OK under normal circumstances, but the machine has already been setup and I have spent so bloody long trying to get this working, that I thought that I would document a small amount of it so that I wouldn't forget how painful this has been.

The first problem that I encountered was that the documentation passed on by the previous sysadmin listed the server as been @ 10.1.0.2, but I actually found the esx server at 10.1.0.3, which was only found by nmap scanning the port range around it. One of the things that threw me off was the fact that the 10.1.0.2 showed up on the nmap scan as responding on an unusual port, I now think that this is/could be an Intel Lights Out Management system (need to confirm this).
I initially assumed that 10.0.0.2 must be the Virtual Center Management server for vmware (as my boss had said that there was an easy way to clone the virtual machines) ... you know what they say about assumptions ... whoever had set this system up had decided that it wasn't worth installing a Virtual Center to admin the esx server (which is fair enough, as you need to licence it). I installed the Virtual Infrastructure Client on a vmware box on my linux workstation (that is a confusing mouthful), pointed it at the esx server (10.1.0.3) and voila as they say, the virtual machines that our company has magically appeared! Two major things jumped out at me about 4 hours later:

  1. There didn't appear to be an easy way to create a clone of a machine (although there did appear to be some preconfigured images there
  2. & the console to view the different servers didn't work

In my usual way, I decided to tackle things ass about. After some judicious googling, I found that the way to create images for the esx server without a virtual center is with a tool called vmkfstools ( http://www.vmware.com/community/message.jspa?messageID=575065). There was only one question - where the hell do I run this mythical command? I spent a fair amount of time in the Virtual Infrastructure Client just poking around, seeing if I could make stuff run from there until I was pretty sure that I'd wasted enough time. On a whim, I decided to go over the nmap logs again, and to my suprise I noticed that the esx server was running a http & https server along with a ssh server - this must be it. I tried ssh'ing directly into it using the only account that I had (the root account), but the guy who'd set this up had been cagey and done what I would have - disabled the root account. I then tried accessing stuff via the web browser, but found that there was slightly less functionality in that than the Virtual Infrastructure Client and the remote console _still_ didn't work!
The next day, I logged back into the Virtual Infrastructure Client and noticed the Users and Groups tab under the root host - I added an account to this, gave it shell access and I was in! I immediately tried the vmkfstools command, but found that it was not there, and I spent a couple of hours investigating the server. In the end, I found the tools under

[abowden@esx sbin]$ pwd
/usr/sbin
[abowden@esx sbin]$ /usr/sbin/vmkfstools

The vmware images were relatively easy to find in

[abowden@esx sbin]$ cd /vmfs/volumes/
[abowden@esx volumes]$ ls
3c323653-70542b88-99be-0007e93378ac  ds01
3c745d5e-63dc1a0a-0dae-0007e93378ac  ds02
44d8935f-ad4981aa-fc75-0007e93378ac  storage1
[abowden@esx volumes]$

And I created my first clone of an image like this:

[abowden@esx volumes]$ su
Password:
[root@esx volumes]# cd /vmfs/volumes/ds02/
[root@esx ds02]# mkdir tmp1
[root@esx ds02]# /usr/sbin/vmkfstools -i \
[root@esx ds02]# /vmfs/volumes/storage1/windows2003_master/windows2003_master.vmdk \
[root@esx ds02]# tmp1.vmdk

The console notified me that it was cloning and after a while it came back.

I then logged in to the Virtual Infrastructure Client and looked for the new image, just to realise that the esx server didn't know about it. It was a simple thing to create a new image, and to point it at the new image which had been created. Unfortunately, I then hit another snag. I could power the thing on but I had no way of knowing whether or not anything was happening. I tried a number of things such as hitting my head against the wall, stabbing myself repeatedly in the hand and trying to the use the Virtual Infrastructure Client/Web client, they were all as effective as one another!

I decided that because the esx server is sitting on what is effectively a cut down linux box, the logs might tell me something and surely enough they did. The log that I found interesting was (suprisingly enough) the secure log:

[root@esx log]# tail -f secure
Jun 27 09:38:11 esx vmware-authd[13480]: Found vmkauthd: /usr/lib/vmware/bin/vmware-vmkauthd-start
Jun 27 09:38:11 esx vmware-authd[13480]: Redirected mks to 10.1.0.3:903
Jun 27 09:38:46 esx xinetd[1082]: START: vmware-authd pid=13481 from=10.0.2.12
Jun 27 09:38:46 esx vmware-authd[13481]: login from 10.0.2.12 as 5288ebbc-7770-1366-ee0b-21d00c324d5f
Jun 27 09:38:46 esx vmware-authd[13481]: Found vmkauthd: /usr/lib/vmware/bin/vmware-vmkauthd-start
Jun 27 09:38:46 esx vmware-authd[13481]: Redirected mks to 10.1.0.3:903

Every time I tried to connect the remote console to the server, I got this repeated set of failures but what struck me was that it was trying to listen on port 903, which didn't have anything listening on it. After some googling, I found the following site ( http://www.vmware.com/community/thread.jspa?threadID=46632 ), in which one of the developers (kitcolbert) notes:

Here's one thing you could try:
- ssh to your ESX box
- add the following to /etc/vmware/config:

vmauthd.server.alwaysProxy=TRUE

- try reconnecting

Hopefully that should allow you to work around the problem.

he goes on to say

Sorry, I should have been more upfront about this: the remote console not working is definitely a bug! The workaround I gave above is exactly that: a workaround. We didn't document it because we didn't think anyone would need it. Unfortunately that's not the case. Be assured that we are investigating this issue.

One question: do you (or anyone else on this thread) have any NAT'ing going on between the remote console and the ESX machine? There is a known (and understood) issue for that. However, if you simply have two subnets connected by a router, then that would be a new issue. Is there anything interesting or fancy about the routing? The reason I ask is that I believe we have people here in-house connecting the remote console across subnets to the ESX server. Thanks.

After I added this, I tried to reconnect and it worked!!! esx logs now say:

Jun 27 09:43:23 esx xinetd[1082]: START: vmware-authd pid=13486 from=10.0.2.12
Jun 27 09:43:23 esx vmware-authd[13486]: login from 10.0.2.12 as 52571b6a-f310-16a3-73c2-15536df93013
Jun 27 09:43:23 esx vmware-authd[13486]: Local connection for mks established.
Jun 27 09:49:45 esx xinetd[1082]: START: vmware-authd pid=13487 from=10.0.2.12
Jun 27 09:49:45 esx vmware-authd[13487]: login from 10.0.2.12 as 52eb6b50-8155-7936-8a61-5bf44192d90c
Jun 27 09:49:45 esx vmware-authd[13487]: Local connection for mks established.

To the best of my knowledge, this is a problem when you're trying to connect across different subnets.

<Unrelated Update>
I had troubles adding an extra disk to the esx server, it wouldn't let me create a disk greater than the default one set (in my case it was 3Gb). It turns out that it's a bug that can apparently be gotten around by rebooting the physical server ( here ), but as that gives me the willies, I went the approach mentioned in that document and did a:

[root@esx cwmedia03-testing]# /usr/sbin/vmkfstools -c 40g disk01.vmdk

and then attached it as a new disk.
</Unrelated Update>

<Unrelated Update #2>
Just a quick explanation of the network devices in vmware:

root@yancy:/vmfs/OS# ls -lah /dev/vmn*
crw------- 1 root root 1190 2007-03-06 13:43 /dev/vmnet0  #bridged (first adapter found)
crw------- 1 root root 1191 2007-03-06 13:43 /dev/vmnet1  #host only adapter
crw------- 1 root root 1192 2007-03-06 13:43 /dev/vmnet2  #bridged, second adapter found
crw------- 1 root root 1198 2007-03-06 13:43 /dev/vmnet8  # NAT adapter, this allows up to 32 connections on it.
root@yancy:/vmfs/OS#
root@yancy:/etc# grep -r 'eth0' vmware/
vmware/locations:answer VNET_0_INTERFACE eth0
vmware/locations:answer VNET_0_INTERFACE eth0

So it looks like it's assigned in /etc/vmware/locations
</Unrelated Update #2>

Popularity: 100% [?]

Everybody's a critic WTF?? Nothing about this made senseas useful as a blindfolded monkey throwing dartsmediocre ... at bestsolved my problem but needed modificationspectacular.  \'Nuff said (3 votes, average: 4.67 out of 5)
Loading ... Loading ...

Leave a Reply

 

 

 

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>