Click near text search result

Hi all,

I’m just getting to grips with the software, and am very impressed so far. One thing I have noticed, though, is that there are fewer actions available when searching for text on a SUT.

I was wondering if there is any way to search for a text string, and then click on a spot relative to the result? This can be done with image searches by moving the hot spot, but I can’t see any similar possibility for text. Is it just not possible?

Thanks,

Gary

For this, if you do not want to use an image search, you can use the OCR (Optical Character Recognition). If you know how where you want to click relative to the text, you can use code like this:

if imagefound(text:"Hello!", SearchRectangle:("Image1","Image2"))
then
	click foundimagelocation()+(100,100)
	log "clicked it!"
else
	logerror "didn't find the word"
end if

To learn more about ways to improve the accuracy of your OCR search please visit this knowledge base post on our website: http://www.testplant.com/support/2012/02/working-with-ocr/

-Elizabeth

Hi Elizabeth,

Thanks for the quick response. By complete coincidence, I have also just figured out a (less elegant) solution:

Click (ImageLocation(text:"Search Text") + (0,20))

I’m not sure how it compares to what your search does, but both fit the bill for me.

Thanks :slight_smile:

Gary

The trick with this is getting the actual size of the word/text. Since you’re not starting with an image, you may not know how many pixels from the found image location (which will be at the center of the identified text) you need to move. This will be especially true if you are using the OCR for its intended purpose, which is finding or reading dynamic text strings. If you want to click to the right of “team” on one run and the next run you need to click next to “association” then you’ll need to come up with an algorithm to make the appropriate adjustment.

There are some built-in functions that can help with this, including the imageRectangle() and imageHotSpot() functions. So if you want to click 10 pixels to the right of any found word, you could use a command like this:

on clickToTheRight myTxt
	put ImageRectangle(text:myTxt) into bounds
	MoveTo(item 3 of bounds + 10, average (item 4 of bounds, item 2 of bounds))
end clickToTheRight

Which could then be called like this:

clickToTheRight "walrus"

The command finds the specified text and gets the coordinates of the bounding area. Then it moves to a point 10 pixels beyond the largest x value and between the top and bottom y coordinates.

That’s an excellent point about working with dynamic text. When working with rectangles there are some SenseTalk functions that return specific points on a rectangle that can make your life even easier. So this could be written as:

on clickToTheRight myTxt 
   put ImageRectangle(text:myTxt) into bounds 
   Click rightCenter of bounds + (0,10)
end clickToTheRight

The nine “interesting” points of a rectangle can each be accessed by name: topLeft, topCenter, center, etc. as well as the top, left, bottom, and right values.

Hi,

Sorry to bring this post back from retirement. I’ve tried to take the suggested solution above and extend it further, but I run into problems when I execute the script.

It runs fine up until the point that it tries to click relative to the image where it proceeds to bail out with the following.

Thanks in advance,
Deon.

Error -
SenseTalk Runtime Exception Invalid operands for addition: expected numbers, dates and/or lists; got ‘(0,35)’ and ‘’

It basically gets to this section and fails, any help figuring out what has gone wrong would be greatly appreciated.


	if direction is "up" then
		Click edge of bounds - (0,abs(distance))
	else if direction is "left" 
		Click edge of bounds - (abs(distance),0)
	else if direction is "down" 
		Click edge of bounds + (0,abs(distance))
	else -- right
		Click edge of bounds + (abs(distance),0)
	end if

Called like -
Toolbox.clickRelativeTo “anImage”, down, 35

Full Script -


on clickRelativeTo anElement, direction, distance, optionalFrom
	(*********************

Simple script which allows you to click a distance from the edge of a rectangle which bounds
an element, either image or text.  The rectangle edge selected is based on the direction
specified.  Up is Top, Left is Left, Down is Bottom and Right is Right.
By default the center of each edge is used as the start or From point.  The From point can
optionally be specified, either left or right for Horizontal edges; top or bottom for vertical edges.
As we are specifying a direction, distance should always be a positive number so in all instances
the absolute value of distance will be used.
Will LogError when attempting to click on, or within, anElement's rectangle.  Due to the absolute
value issue above this will only be possible if a distance of zero is used.

Parameters :

anElement : the Image, or Text,  you want to click relative to
direction : "up", "down", "left", "right"  (Not case sensitive)
distance : Number of pixels in "direction"
optionalFrom - 
Optional field.  If direction is "up" or "down" then only "left" or "right" can be used.
If direction is "left" or "right" then only "top" or "bottom" can be used.
If this field is left empty then "center" is used.

      *********************)
	
	if distance is 0 then
		LogError("Distance must not be zero")
	end if
	
	try
		if imageFound(anElement) then
			put ImageRectangle(anElement) into bounds
		else
			put ImageRectangle(text:anElement) into bounds
		end if
	catch e
		LogError("Error finding image or text["& anElement &"] : ", e)
	end try
	
	
	put ("up", "down") into validVerticalDirections
	put ("left", "right") into validHorizontals  -- Used for both direction and optionalFrom
	put ("top", "bottom") into validVerticalFroms
	
	if direction is "up" then
		put "top" into edge
	else if direction is "down" then
		put "bottom" into edge
	else if direction is in validHorizontals then
		put direction into edge
	else
		LogError("Invalid direction [" & direction & "]")
	end if
	
	if optionalFrom is empty then
		put edge & "Center" into edge
	else 
		if ((optionalFrom is in validHorizontals and direction is in validVerticalDirections) or \
				(optionalFrom is in validVerticalFroms and direction is in validHorizontals)) then 
			put edge & optionalFrom into edge
		else
			LogError("Invalid combination of direction and optionalFrom produces [" & edge & optionalFrom & "]")
		end if
	end if
	
	Log("Direction is [" & direction & "]")
	Log("Edge is [" & edge & "]")
	Log("Distance is [" & distance & "]")
	
	if direction is "up" then
		Click edge of bounds - (0,abs(distance))
	else if direction is "left" 
		Click edge of bounds - (abs(distance),0)
	else if direction is "down" 
		Click edge of bounds + (0,abs(distance))
	else -- right
		Click edge of bounds + (abs(distance),0)
	end if
end clickRelativeTo

Ok, it looks like you’re carefully and cleverly constructing the name of the function that you want to call in the variable edge and trying to call that. But your code currently tries to call the “edge” function instead. To use the contents of a variable as the name of a function you have to enclose it in parentheses so SenseTalk will evaluate it as an expression:

Click (edge) of bounds - (0,abs(distance))

That should do what you want. Just a hint from an old-timer, though: sometimes simple is better (and easier!) than clever. In this case I think I would just do it this way (since your command is different in each case anyway):

if direction is "up" then 
      Click topCenter of bounds - (0,abs(distance)) 
   else if direction is "left" 
      Click leftCenter of bounds - (abs(distance),0) 
   else if direction is "down" 
      Click bottomCenter of bounds + (0,abs(distance)) 
   else -- right 
      Click rightCenter of bounds + (abs(distance),0) 
   end if

Good luck!

Thanks, worked like a charm.

I agree re: simplicity, it was my first foray into writing something re-useable and I may have been a little lost in the moment. :lol:

Thanks again,
Deon.

Understood. :slight_smile: And I applaud your effort! It’s a lovely handler, and putting in the effort to make things reusable like that can really pay off down the road.

I’m wondering whether it is possible to devise a script which will search a webpage for a certain string of text, and then click the link in the element id …Previewing pages; Understanding an individual search result; click test Using an entire results page. This lesson looks at how to read a search … Hey all, I need some help with a roadblock I have been having. I am using UiPath to find text on a webpage and click a checkbox next to it.