Solaris10 - LDOM - unable to login

LDOM_interface_not-working-ssh-not-responding

User complain, they can't login,
I tried to login, but I could not login. Ping not responding. Login through the console, and server was on up state. Checked ssh service, ok and running. interface up.
ping gateway, good. tried to ssh to outside, I can ssh to control domain and other server on the same subnet.

Checked the subnetmask and it was different. Nothing has ever been changed. Server is up for 66 days, nothing changed.
Surprise... I assigned the ip with correct mask and I am able to ping, and ssh to it. But it keep timeout after login into in less than 30 second. no message says what
really happened.
restart network and ssh, it start disconnecting in between 1-2 minutes. Interesting. Checked configuration on Control domain, interface and other virtual server on the
same interface are working just fine.
Requested to reboot the server.
server came up fine with correct ip and subnetmask but issue still the same. keep disconnecting within 2 minutes.
Went to control domain and assign a new interface from different physical interface. Pointed out the ip to new interface and started network services. It simply start
working fine since then. Interesting ....

1. Verify netmask info
# cat /etc/netmask
#
#
192.168.102.0    255.255.255.128
2. You have wrong netmask of ff000000
[root@sdmr-mw-v02]# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vnet0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 192.168.102.104 netmask ff000000 broadcast 10.255.255.255
        ether 0:14:4f:fa:c2:63
3. Reassign ip address Using correct mask
[root@my-ldm-v02]#  ifconfig vnet0 192.168.102.104 netmask 255.255.255.128 up
[root@my-ldm-v02]#
[root@my-ldm-v02]# ssh 192.168.102.105
ssh: connect to host 192.168.102.105 port 22: Connection refused
[root@my-ldm-v02]# svcs -a | grep ssh
online         Feb_18   svc:/network/ssh:default
[root@my-ldm-v02]# svcadm refresh  svc:/network/ssh:default
[root@my-ldm-v02]#  svcs -a | grep ssh
online          9:29:50 svc:/network/ssh:default
[root@my-ldm-v02]#
4. Now, shows the correct subnet mask.
[root@my-ldm-v02]# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vnet0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 192.168.102.104 netmask ffffff80 broadcast 192.168.102.127
        ether 0:14:4f:fa:c2:63
[root@my-ldm-v02]# Apr 25 09:31:55 my-ldm-v02 sshd[29729]: fatal: Read from socket failed: Connection re                                                                  
                                                                set by peer
[root@my-ldm-v02]# ls -l /etc/netmasks
lrwxrwxrwx   1 root     root          15 May 17  2013 /etc/netmasks -> ./inet/netmasks
[root@my-ldm-v02]# Apr 25 09:34:54 my-ldm-v02 sshd[29597]: monitor fatal: Timeout before authentication
Apr 25 09:39:18 my-ldm-v02 sshd[29693]: monitor fatal: Timeout before authentication for 172.16.250.223
[root@my-ldm-v02]#
[root@my-ldm-v02]# svcs -a | grep -i physical
online         Feb_18   svc:/network/physical:default
[root@my-ldm-v02]# svcadm restart  svc:/network/physical:default
[root@my-ldm-v02]# Apr 25 09:42:21 my-ldm-v02 sudo:  kbhusal : TTY=pts/4 ; PWD=/export/home/kbhusal ; US                                                                  
                                                                ER=root ; COMMAND=/usr/bin/su -
Apr 25 09:42:21 my-ldm-v02 su: 'su root' succeeded for kbhusal on /dev/pts/4
[root@my-ldm-v02]#
[root@my-ldm-v02]# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vnet0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 192.168.102.104 netmask ffffff80 broadcast 192.168.102.127
        ether 0:14:4f:fa:c2:63
6. Still problem to ssh to the host. rebooting
[root@my-ldm-v02]# init 6
[root@my-ldm-v02]# svc.startd: The system is coming down.  Please wait.
svc.startd: 85 system services are now being stopped.
syncing file systems... done
rebooting...
Resetting...

7. After reboot, verify ip/mask.
[root@my-ldm-v02]# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vnet0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 192.168.102.104 netmask ffffff80 broadcast 192.168.102.127
        ether 0:14:4f:fa:c2:63
[root@my-ldm-v02]#

[root@my-ldm-v02]# netstat -in
Name  Mtu  Net/Dest      Address        Ipkts  Ierrs Opkts  Oerrs Collis Queue
lo0   8232 127.0.0.0     127.0.0.1      1894   0     1894   0     0      0
vnet0 1500 192.168.102.0  192.168.102.104 18369  0     4788   0     0      0


8. Still problem, Now decided to add new vnet to the LDOM from different interface card.
Login to control domain add a vnet - NIC
Added new interface from different interface
[root@physical_serv-p01]# ldm add-vnet vnet2 primary-vsw3 my-ldm-v02
[root@physical_serv-p01]# ldm rm-vnet vnet2 my-ldm-v02
[root@physical_serv-p01]# ldm add-vnet vnet1 primary-vsw3 my-ldm-v02
[root@physical_serv-p01]#

9. Now, Login to LDOM and bring down the old interface and bring up the new one.
[root@my-ldm-v02]# ifconfig vnet1 plumb

[root@my-ldm-v02]# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vnet0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 192.168.102.104 netmask ffffff80 broadcast 192.168.102.127
        ether 0:14:4f:fa:c2:63
[root@my-ldm-v02]# ifconfig vnet1 plumb
[root@my-ldm-v02]# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vnet0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 192.168.102.104 netmask ffffff80 broadcast 192.168.102.127
        ether 0:14:4f:fa:c2:63
vnet1: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
        inet 0.0.0.0 netmask 0
        ether 0:14:4f:f9:92:70
[root@my-ldm-v02]# svcs -a | grep physical
[root@my-ldm-v02]# svcadm restart svc:/network/physical:default
[root@my-ldm-v02]# ifconfig vnet0 down
[root@my-ldm-v02]# ifconfig vnet1 up
[root@my-ldm-v02]# mv /etc/hostname.vnet0 /etc/hostname.vnet1
[root@my-ldm-v02]# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vnet0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 192.168.102.104 netmask ffffff80 broadcast 192.168.102.127
        ether 0:14:4f:fa:c2:63
vnet1: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
        inet 0.0.0.0 netmask 0
        ether 0:14:4f:f9:92:70
[root@my-ldm-v02]# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vnet0: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 192.168.102.104 netmask ffffff80 broadcast 192.168.102.127
        ether 0:14:4f:fa:c2:63
vnet1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
        inet 192.168.102.104 netmask ffffff80 broadcast 192.168.102.127
        ether 0:14:4f:f9:92:70
[root@my-ldm-v02]#
[root@my-ldm-v02]#
[root@my-ldm-v02]#  ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vnet0: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 192.168.102.104 netmask ffffff80 broadcast 192.168.102.127
        ether 0:14:4f:fa:c2:63
vnet1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
        inet 192.168.102.104 netmask ffffff80 broadcast 192.168.102.127
        ether 0:14:4f:f9:92:70
[root@my-ldm-v02]#


LOGINFROM=`who am i | cut -f2 -d"(" | cut -f1 -d")"`
# ifconfig vnet0 192.168.102.104 netmask 255.255.255.128 up