It really sounds like you may have two different versions of Edge installed, you can confirm this from programs and settings. If you uninstall the application then the icon on your toolbar should have disappeared. Uninstall all versions of edge and if you still have that icon on your toolbar, try clicking it and see what happens.
The two different versions are outlined in the following article, did you receive the new version from windows update or did you install it yourself? Either way I believe if you follow this guide to the letter, reboot and manually install then you will have sorted out the issue.