Mouse pointer 'set position' or bust!

There has been a lot of talk about WebGL as a new platform for games. However, JavaScript lacks one feature that may prevent it from ever competing with the big boys. Unless there is some method unknown to me, a free-look camera that follows the movement of the mouse pointer, such as what you’d find in nearly every first-person AAA game, cannot be duplicated in JavaScript because you have to be able to set the position of the mouse pointer. This capability has been left out because of the clear security risk, but still…

Solutions?

It’s certainly a serious pain…but I don’t think you can overcome it - and I’m fairly certain that the browser writers will never provide you with that feature.

The problem is that the bad guys could either trap your mouse inside a series of advert slide-shows - creating the spam from hell - or they might be able to insert a mouse movement command into some content like a google advert and thereby trick you into “clicking” on something you didn’t want to click on.

Imagine this:

  1. Hide the mouse cursor.
  2. Reposition the mouse cursor 100 pixels further down the screen.
  3. Draw a fake mouse cursor where the user thinks it should be.

Now you can control what he clicks on, where he types his passwords…you name it.

There are a whole host of sneaky exploits you could do.

But it is a SERIOUS pain for people wanting to write “conventional” games like 1st person shooters…since you have no way to prevent the mouse from moving out of your window - you can’t stop the player inadvertantly clicking on something just outside of your window in a fit of high-speed action and killing the game or doing something else rather nasty.

…and forget 1st person “mouse look” - there is no hope of being able to do exactly that.

I think we’re going to have to look into the prospect of going back to game-pads and keyboards as the main input devices…with the mouse relegated to menu clicking and such.

It’s definitely not going to be an easy transition.

What is the big deal then?!

I just came aware of that problem since I try to push my own app. What is the plan to solve this issue? When will this issue be solved? This is actually ridiculous that WebGL is told to be the new hot stuff for 3D and games around the net - and then it cant even maintain the most basic game-features?!

Please tell me, anyone, how this will be solved or I can see no reason why WebGL should be suitable for game-developers presenting their apps… I can see more future for the “Adobe´s molehill API” instead…

Also: why would adobe not include/attach their flash-player to all browsers?! Should be the same like WebGL - but faster since JavaScribt does not seem to be a suiteable programming language…

overview required

Thx
KK

I don’t think it’s a transition people will accept, at least not for FPS-type games. I am certainly aware of the potential exploits, but the fact is there are things that can be (and are) exploited now. The only solution I can think of is some kind of security enabling, where features are made available only if the site is somehow marked as a trusted one.

Thanks lthornblad, everyone…

  1. what exactly is the problem why this wont be possible that easy?
  2. Why would it be a security problem?

When this will be possible as you mentioned (to mark secure sites) I can at least see the point. The Khronos-Guys have to improve on that. I wonder what else wouldn’t be possible. And I wonder what disadvantages the “molehill API” has… Otherwise I can see everybody fall back to flash (which will have improved to be suitable for 3D - obviously better than WebGL)

  1. May it be that ActionScript is the better (faster) programming language compared to JavaScript?
  2. I assume you wouldn’t have that “mouse pointer-issue” under ActionScript?!

Thx
KK

  1. Would you have the same issue under ActionScript?! I tried it quickly and it appears to be the same… May it be that the stronger molehill API is limited to the same restrictions?!

I mean we are talking about the webs future in gaming - so far I cant see this happening. There will be a lot of good games even without the FPS-style… But still: there are heaps of apps depending on this. At least tell us it wont ever happen so we can work around it :frowning:

Reg
KK

I explained why it’s a security issue.

There are evil bad guys out there.

Suppose an advertiser makes a popup advert with embedded JavaScript…if they can reposition the mouse pointer - they can completely prevent you from clicking anywhere other than on their advert - they can prevent you from typing into another window - they can stop you from backing out of the web page…or even shutting down the browser! Most users will simply resort to rebooting the computer.

That’s completely unacceptable - so we don’t give JavaScript applications the right to do that. If you know of a way to avoid bad guys getting onto the Internet - that would be good - but all the while they’re here - we have to put up with these kinds of restriction in the name of not having these bozo’s screwing up our lives.

Actually - I believe that there are even NASTIER security implications - they could (for example) turn off the mouse cursor - clamp the mouse to the middle of the screen - but track how the user tries to move it and draw a FAKE mouse cursor at that location. You might think you were clicking on some button - when in fact you weren’t. I bet a smart person could figure out a way to turn that into a hack that would let them steal your credit card number or something. It’s just too dangerous.

Sure, it’s a pain for games - we’re going to have to find another paradigm. But that IS how it is - and it’s NOT likely to change. Even if you did find a sneaky loophole way around it (which is unlikely, IMHO) - the browser writers would regard it as a bug and pretty soon, they’d “fix it”.

So there is no point getting upset about it - you just have to start thinking of more creative ways to control your game. We had games before the mouse was invented - most console games don’t use a mouse - there are ways. Adapt, improve.

Thanks for making this more clear to me, Steve…

Now I understand much better. So is this a pure JavaScript-issue? Would it be the same under ActionScript?

The thing is that this feature is indispensable for some games. How else would you play shooters like Counter Strike… There isn´t a more convenient solution as the way we used to play games like this for years… Its also the most intuitive control. A game controller would do the job. But it is not very user-friendly to expect everyone has one.

So yeah you might work around it in most cases - but when it comes down to shooters (and you dont have a pad): it is going to be such a pain!!

Thats why my question is: will it be the same under AS?

Please note that some devs rely on those statements before pushing every effort into development turning out to be not suitable for this specific job.

Thx
KK

@Steve - Sure, we can live with it and find other ways, but they would hardly be ideal. I won’t play FPS-style games on a console.

Although I have not tried, it seems quite possible to move the entire browser around to match the mouse position, thus preventing someone from clicking wherever they want. Even if that is not possible, there are always exploits. Hopefully, as the demand for quality web-based 3d apps grows, browser makers will find a solution.

@King - AS is Flash-only, I believe. Its use is more limited for those of us looking to make cross-platform apps. I don’t know if it has the feature or not, but it could be exploited just the same. Perhaps that is why Apple refuses to support it.

Well I investigated this a bit: Yes, it will be the same under ActionScript. Which pretty much means there wont be any reasonable shooter coming up into the browsers for the next few years. Its crap but valuable to know - someone should have mentioned that earlier^^

Anyway I thought of another way to do it. It uses (invisible) overlays (movie clips) which interact with the mouse pointer. Like this: [http://]activeden.net/item/fullscreen-background-fully-customizable-resizable-floating-background/full_screen_preview/30451?ref=prowebmedia

To implement this in 3D would still not be ideal for FPShooters… But we are getting there. The idea is to more conveniently pan the view instead of click&drag the camera. Let me know when you can think of a better way to control the camera in space.

thoughts welcome

When somebody runs into this, would you mind to share the source code? I´ll be trying to implement this. But using other engines + I dont have too much time to get it done

Thx
KK

The thing that the “activeden” demo does is trivially easy - however, it does not (and cannot) do what PC-based FPS’s do with the mouse. You can pan across the field of bubbles for a while - but when you reach the edge of the screen - you can’t turn any further in that direction. It’s using the rate of change of position of the mouse like an FPS does - but because it can’t reset the cursor to the middle of the window - you keep running out of screen.

There is no doubt that you could implement a scheme that panned or tilted the camera by an amount that’s proportional to the distance from the center of the window - but that leaves you with the “running out of screen” problem.

You could get around that by moving the camera at an angular velocity that’s proportional to that distance of the mouse from the center. When the mouse is in the middle of the window - the camera stays still - the more you move the mouse away from the center, the faster the camera moves. This certainly gets around the “running out of screen” problem.

But the problem with that (trust me - I’ve tried it) is that it’s hard for the player to stop the camera exactly where he wants it to be because it’s not clear where the “center” of the screen is. You can’t find the “null point” easily enough when the action gets hot. This problem is 100 times worse in a classic FPS where the mouse controls not just the camera - but also the direction you’re moving in. With a velocity-control system, you just can’t get yourself pointed in the right direction reliably enough and you wind up lurching around like a drunkard!

However, in a console-based game, that’s exactly how the joystick works. The further the joystick is moved away from the center, the faster the camera turns. But the only reason that works well is because joysticks have a physical spring that returns them to the center position when you release them. So you can stop moving at any time by just letting go of the joystick.

The mouse doesn’t have that - so in conventional FPS’s, you have to rely on RELATIVE mouse movement - not on it’s absolute position. That means that when you stop moving the mouse, the relative motion goes to zero automatically and the camera stops moving - which is like releasing the joystick and letting the spring recenter it.

In effect, you’re mapping the velocity of the mouse to the velocity of the camera…or (if you prefer) the position of the mouse to the position of the camera…but that’s the problem. If you use the position of the mouse - then you’re limited to positions that lie inside the browser window…and now you can’t keep on spinning the camera around without “running out of screen”.

So the mouse is basically a bust…we can’t map mouse position onto camera velocity and we can’t map mouse position onto camera position either.

Using the keyboard arrows has some of the same properties as a joystick or relative-mouse-movement system - when you release the keys, you’re returned to the “null” position. In a sense, your finger pressure on the key maps to camera velocity - and releasing your finger makes that go instantly to zero.

But the trouble with keyboards is that they aren’t proportional devices - a key is either down or it’s not - so camera speed becomes a binary property. You can’t control the speed at which you move the camera. Either it’s too fast and the world swirls past dizzingly while you’re looking carefully for something in your visual field - or it’s too slow for a ‘quick look’ to your right or left and you feel like your head is stuck in treacle!

But it’s worse even than that. Doing even that little with the keyboard is a nightmare because of N-key rollover issues. Most PC keyboard hardware can’t reliably detect more than two keys held down at the same time…unless those keys are selected very carefully - AND with a knowledge of the underlying keyboard switching matrix that varies from one keyboard to another in ways that are impossible to detect automatically.

Try this…hold down any two keys and start pressing other keys nearby…pretty soon you’ll find that some don’t register. On my beloved GoldTouch keyboard, if I hold down (say) ‘A’ and ‘C’ then the ‘S’ and ‘F’ keys still work - but the ‘D’ key doesn’t. But on my Dell Laptop, ‘D’ works just fine but ‘F’ doesn’t! That’s a hardware restriction - and it’s very difficult to work-around because there are a bunch of different keyboard controllers out there in the world, and you can’t query which you have - and they aren’t compatible.

This makes two-handed keyboard play decidedly problematic. If someone is using W,A,S,D to move around and other keys to drive camera motion - then they will interfere with each other and you’ll find that if you (say) try to move diagonally by holding down W and D together - that suddenly some other random key will stop working and maybe you can’t jump anymore!

So this is rather a fundamental problem.

In an ideal world, we’d be able to stick an analog joystick onto every phone and keyboard or issue game-pad controllers to everyone who plays an online game…but that’s not the kind of thing that can happen overnight. Worse still, the world is leaping forward in the direction of touch-screens - and that makes things MUCH worse because it eliminates even the possibility of repositioning the cursor!

If online 3D gaming truly takes off on the PC - then I think we should expect to see the return of the USB gamepad. With people playing games on iPad’s and the like, they aren’t going to want to have their fingers covering up the action - so the touch screen is a non-solution as an input device…so there is a problem there too.

Using tilt sensors in cameras and pads has a similar problem to the mouse - you have to map tilt angle onto camera velocity - and finding that null position is like balancing a ball on a book!

For 3rd person games, I think the answer is fairly clear - position the mouse (or your finger) on top of your avatar to stand still - move it further away in the direction you want to move to start the guy walking - the larger the distance, the faster he moves. We’re using mouse position to control avatar/camera speed…but because your avatar is a focal point for your attention, it’s easy to see how to stop the motion. It’s not ideal - but it does work. Another nice thing about 3rd person games is that camera control is MUCH less critical. In an FPS, you need the two-handed style where one hand is on the mouse - controlling the camera and the direction you’re walking in - the other controlling the speed of movement and allowing you to step sideways and backwards.

Thanks for your thoughts… And Jesus that was a marathon. Thanks for your effort.

The demo you have seen was of course a 2D sort of thing. But applying this to 3D-space you would not have the border issue (as you would just keep spinning around - you just named it yourself). So far I still think this is the best basic approach. It is not meant to be the same like FPS´s - but to more conveniently pan the view. All your statements have been true (so far I can tell) and I agree about that “null”-issue. It might not be very user-friendly once it struggles to tell the null-point. So this got to be investigated. Maybe by applying a bigger tolerance for the null-point might let you keep the control. There is got to be something to do to improve on that… I can see no other option to this. Users (like me) wont ever play FPS just on a keyboard (even if they could). Moving just by mouse (via button-overlay) or by touching screens should be additional options - not the only available control-style.

This is not even affecting just FPS-games (3rd-Person-Shooters as well^^). Think of it as a more convenient solution to just walk around in 3D-space without necessarily holding a gun in your hand. How else would you explore 3D-environment? Surely not by crappy ways to control your avatar. People would more likely refuse this as trying to cope with that.

I am not trying to give you a hard time - further I thank you for your honest reply. I am just a little concerned about this. And I wonder whether its just the few of us wondering about the future of conveniently exploring the 3D-space…

I think this will be worth investigating. In case we fail - I am highly interested what would be plan C. So thanks, Steve and all the others don’t hesitate to speak your mind.

Reg
KK

The demo you have seen was of course a 2D sort of thing. But applying this to 3D-space you would not have the border issue (as you would just keep spinning around - you just named it yourself).

I disagree - it doesn’t matter in the slightest whether it’s 2D or 3D. In that demo - when you move the mouse out to (let’s say) the extreme left side of the window - the sideways motion of the bubbles stops - and now you have no way to move any further to the left. If this were in 3D, you might see a doorway off to the left there where you want to go…and you can’t get there because you’ve run out of window.

I thought of a sensible solution, but it is not how browsers work now, so it is purely theoretical.

Because of touch devices, any browser-based interface should only rely on two types of mouse actions: 1. some point is clicked, 2. a drag starts, goes through some intermediate points, and ends.

Now, if an user starts a drag action in browser window using a mouse, and if mouse is dragged outside the window, it could still be sending events to the page (until the drag ends), even with negative coordinates.

It poses no security risk and is compatible with touch devices (when a touch device user hits the screen border, he has to release and start another drag to pan further).

So in case of a FPS, shooting would have to be implemented as a keyboard-driven, and the user would have to keep a mouse button pressed all the time, but other than that, it would work as expected on desktop computers.

@Steve: I can think of it this way: You have a camera in space. Imagine to apply actions to the camera - call this action through (an invisible) 2DOverlay/Button. So you might have 4 buttons on the screen. One for each direction. Now apply the action to the button which changes the rotation of the camera by a vector. The further you move away from center - the faster the cam spins. So even if you spin real slow, you would actually keep on spinning this way. Haven´t tried it yet. But it sounds plausible to me…

@kguciek: we have to do better than that. I checked the control-style you mentioned before. It would still be a pain for FPS´s. This might be the best we have so far but it is not very intuitive nor user-friendly + unless there wont be some improvement you will experience that its barley capable of handling keyboard and mouse input.

So there is still a fair bit of work to do. Lets stay in touch once anybody has a possible solution.

Regards
KK

What you are saying (in a rather complicated way) is that you want to map the POSITION of the mouse to the VELOCITY of the camera motion. As I observed before, this is a tricky thing for the user to control because (s)he has to be able to instinctively move the mouse to the center of the screen in order to make the camera stop moving. This is tricky to do while you are concentrating on things other than where the mouse cursor is. In a FPS, you’re looking for bad guys - not watching the mouse pointer.

In practical terms, the issue is “finding the null point” - the place where all motion stops.

It has been suggested that you have a large “dead zone” in the center of the screen to make it easier to hit - and that’s certainly possible (and necessary!) - but in practical use, it still doesn’t solve the problem. What happens is that once you get the mouse just into the edge of the dead zone, and the camera stops moving - the player (who, remember, is doing this on a subconscious level while trying to actually concentrate on the game action) only notices that the camera stopped. Now, the mouse pointer is right on the edge of the dead zone. If he wants to move the camera in one direction, the tiniest of movement will set it moving again - but if he wants to move in the other direction - he has to move the mouse all the way across the dead zone to get it going again. It turns out that this feels really bad.

But no matter how you phrase it or implement it - you’re down to four kinds of basic options:

  1. Mouse velocity maps to camera velocity…what conventional FPS’s do.
  2. Mouse position maps to camera velocity…what you’re proposing.
  3. Mouse velocity maps to camera position…unplayable!
  4. Mouse position maps to camera position…actually, the same thing as (1).

Every scheme you can possibly imagine boils down to one of those four things…and they are the only four…unless you’re going to try:

  • Mouse position maps to camera acceleration.
  • Mouse velocity maps to camera acceleration.
  • Mouse acceleration maps to…etc

But these are even harder to control than the earlier ones.

I suspect that in the end, we’re going to have to go with some variation of (2) - because of touch-screens. But it’s not nice.

– Steve

True! I admit this would not work well for FPS´s. So kguciek mentioned the best solution so far (which still is not very intuitive, though).

What else can be done? Can´t we somehow prevent people to access/manipulate the command “set position”? Any additional ideas? Anyone?!

I have been working with something that “kind-of” works…
It isn’t something that one would hope for - it’s a java applet. But that’s what it is… mouse control with the Robot class in java.

Hmm no way to attach something? Well i guess you guys will have to compile this on your own. Package it into a jar, digitally sign it and make sure it has full permissions. BlueJ (google it) will package it and give it full permissions - at least it did for me. You will still need to self sign it though. Anyway code:

CenterMouseMain.java :

 
import java.awt.Button;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.event.KeyEvent;
import java.awt.event.KeyListener;
import javax.swing.JFrame;
import javax.swing.JLabel;

public class CenterMouseMain extends java.applet.Applet implements ActionListener{

    JFrame frame = new JFrame("CenterMouse");
    JLabel label;
    Button launchButton;
    
    public void init(){ 
        resize(140,30);
                        
        MouseMoveWindow win = new MouseMoveWindow();
        
        label = new JLabel();
        label.setText("<html><font color='blue'><u>Center Mouse</u></font>

Alt+Tab back
to this window
and press Escape
to stop it.</html>");
        win.add(label);

        frame.getContentPane().add(win);
        launchButton = new Button("Run CenterMouse");
        launchButton.addActionListener(this);
        this.add(launchButton);
        
        frame.addKeyListener(new KeyListener(){
            @Override
            public void keyPressed(KeyEvent event){
                if(event.getKeyCode() == KeyEvent.VK_ESCAPE){
                    ((MouseMoveWindow)(frame.getContentPane().getComponent(0))).stop();
                    frame.setVisible(false);
                }
            }

            @Override
            public void keyReleased(KeyEvent arg0){}

            @Override
            public void keyTyped(KeyEvent arg0){}
        });
    }
    
    @Override
    public void actionPerformed(ActionEvent arg0) {
        frame.pack();
        frame.setVisible(true);
        ((MouseMoveWindow)(frame.getContentPane().getComponent(0))).start("time", 100);
    }
}

MouseMoveWindow.java :

package CenterMouse; 

import java.awt.AWTException;
import java.awt.Color;
import java.awt.Dimension;
import java.awt.MouseInfo;
import java.awt.PointerInfo;
import java.awt.Point;
import java.awt.Robot;
import java.awt.Toolkit;
import java.util.Timer;
import java.util.TimerTask;
import javax.swing.JPanel;

public class MouseMoveWindow extends JPanel {
        
    private static final int WIDTH = 150, HEIGHT = 100;
    
    private Timer timer;
    private TimerTask resetMouse;
    private TimerTask resetMouse2;
    int condAmountSquared;
    
    public MouseMoveWindow(){
        setPreferredSize(new Dimension(WIDTH, HEIGHT));
        setBackground(Color.white);
    }
    
    public void start(String condType, int condAmt){
        if(condType.compareTo("time") == 0){
            resetMouse = new ResetMouse();
            timer = new Timer();
            timer.scheduleAtFixedRate(resetMouse, 0, condAmt);
        }else if(condType.compareTo("distance") == 0){
            condAmountSquared = condAmt*condAmt;
            resetMouse2 = new ResetMouse2();
            timer = new Timer();
            timer.scheduleAtFixedRate(resetMouse2, 0, 100);
        }else{
            resetMouse = new ResetMouse();
            timer = new Timer();
            timer.scheduleAtFixedRate(resetMouse, 0, condAmt);
        }
    }
    
    public void stop(){
        timer.cancel();
    }
    
    private class ResetMouse extends TimerTask {
        @Override
        public void run() {
            try {
                Robot robot = new Robot();
                Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
                robot.mouseMove((screenSize.width/2), (screenSize.height/2));
            } catch (AWTException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
    }
    
    private class ResetMouse2 extends TimerTask {
        @Override
        public void run() {
            try {
                Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
                Point pt = (MouseInfo.getPointerInfo()).getLocation();
                int centerX = (screenSize.width/2);
                int centerY = (screenSize.height/2);
                int xDiff = pt.x - centerX;
                int yDiff = pt.y - centerY;
                if((xDiff*xDiff + yDiff*yDiff) > condAmountSquared){
                    Robot robot = new Robot();
                    robot.mouseMove(centerX, centerY);
                }
            } catch (AWTException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
    }
}

P.S.
Sorry for crap coding… I dont normally use java. But I cooked this up to test the concept. Just make sure javascript can tell when the mouse was set back to the center of the screen (not too hard because it happens instantly with each timer tick). I’ve gotten ok results so far… So what do you guys think?

Well, I don’t think there was ever a doubt that you could do it in Java - but most people disable Java on their browsers.

Actually there is a proposal for setting the mouseposition in the mozilla wiki but i can’t seem to find it anymore.

It was a bit like this.

First you needed to request a fullscreen mode and this needed to be approved of by the user.
secondly you needed to request the mouse setting api and this also needed to be approved by the user.
thirdly IF both were granted then you could use them but the fullscreen would have a small bar showing the key to both disable the mouse postioning system and fullscreen.

Also You need to be fullscreen to be able to set the mouse postion