Proposal of new topology information Face-list for manipulation planning of deformable string tying

Our research group has been focusing on the automation of string tying-untying operations by a robot. In this paper, shape abstraction data of a string named Face-list and an operation planning method with Face-list are proposed. In our planning method, string deformation is represented as a state transition of known topology representation P-data and proposed shape abstraction data Face-list. Face-list enables a robot to decide shape operations that can correctly generate a string deformation plan toward the target shape. In addition, an optimal current operation for changing the string toward the target shape is decided based on scoring the difficulty of total shape operations. The results of two experiments using the proposed method are reported in the final.


Introduction
There are many deformable objects, such as string and cloth around our living space.Recently, the demand to manipulate deformable objects, such as string and cloth by robots is growing.The reason is that it has the possibility of making our lives more convenient.There are also many application domains, such as housework, manufacturing, and medical field.Actions to operate deformable objects, such as string and cloth are very common and important when the housework robots work around our living space.In the medical field, various surgical robots have been developed and introduced, but their operations are performed by doctors.If the work of suturing the affected area after surgery can be automated, it will reduce the burden on doctors [1].In addition, in factories, a lot of manipulators are introduced to automate manufacturing processes.However, many processes, which use deformable objects such as the wiring process of flexible cables, are not automated [2,3].The reason is that it is difficult for robots to manipulate the shape of amorphous objects, and it remains an unsolved problem.This is because the operation of deformable objects has problems with shape recognition and shape prediction, and it cannot be operated by using a routine operation, such as teaching playback.Therefore, our research group has been focusing on constructing a system that can freely operate string with an industrial manipulator.
In conventional research on the shape operation of deformable objects, Ladd et al. [4] investigated the planning method for knot untangling using knot theory.Their method was, however, only demonstrated in a computer simulation.Takamatsu et al. [5,6] proposed a representation method for knot-tying tasks using Pdata.They indicated that P-data are useful to represent the topological state of a string and easy to handle by computers.Wakamatsu et al. [7,8] proposed a planning method based on the topology transition of string and a principle to classify the action of manipulator depending on the shape of part of string.The shape operations and the robot motions were, however, manually selected in their experiments because the difficulty of shape operation in actual situations is not considered.Wang et al. [9] proposed a general approach to changing knot geometry to allow tying and untangling using simple motions of a robot arm.However, in their research, a vision sensor is not applied.The robot manipulator follows predefined paths based on the calculated layout without visual feedback.
In recent years, there are also studies that teach robots to learn actions such as tying and untying strings by learning methods.Vinh et al. [10] proposed a motion generation method by analysing human-performed knotting actions.They also conducted a string knotting experiment.It can be said that this method depends on human interventions such as teaching playback.Lui et al. [11] have realized an untying operation by the twin bowl robot.In their study, the behaviour learning method is proposed by using the original evaluation function for scoring the robot motion and applying the research of Jiang et al. [12].Lee et al. [13] have conducted shape manipulation experiments on deformable objects in the study that incorporates learning into the force control of a robot.In their research, experiments, and discussions have been conducted on adjusting the control gain by learning and tightening the string.
In our previous study [14], a planning method of string untying operation based on knot theory and algorithms to generate the motion of a manipulator has been proposed.String untying operation experiment is also conducted and succeeded by applying this planning method.P-data, Reidemeister move, and Cross are introduced in the planning method.P-data are the representation method of the topological state of a string.That is generated from a point chain model, which is a data structure to describe the 3D shape and position of a string proposed in our previous study [15,16].Reidemeister move and Cross are the basic shape operations of a string defined in knot theory originally.
However, when only P-data are used, the deformation of the string up to the target shape in the string knot operation cannot be controlled correctly.In this paper, we propose a new shape abstraction data Face-list and an operation planning method using it to execute a string knot operation effectively.The novel contribution of our method is the automatic selection of optimal shape operation based on a cost function.The transition of topological state caused by both Reidemeister move and Cross is represented as a tree diagram of P-data and Face-List.Our cost function provides scores with each shape operation in the tree diagram.Then, the path that has the minimum total score is selected as the optimal shape operation.The concept of proposed algorithms for robot motion generation is a deterministic approach to automatic motion generation for the string tying operation.Finally, the robot motion plan based on our proposed method was generated, and its effectiveness was confirmed by the knot-tying experiments using a manipulator.

Overview of the string tying operation with autonomous robot
As shown in Figure 1, in this research, the string tying operation is divided into six phases to realize the string tying operation by the robot.Firstly, the shape of the string is acquired using a distance camera.Secondly, the point cloud data will be output from the camera's raw data.Then, the point chain model of the target string is generated by the combination of the thinning process of the point cloud data and the matching method proposed in our previous research [15,16].The purpose of planning of shape operation phase is to determine the optimal shape manipulation to apply to the string.Using the knowledge of knot theory, the change in the knot state is calculated by a computer after performing a shape operation.A series of shape operations that change the string from the current shape state to the target shape state is derived.Furthermore, the optimum shape operation is determined by scoring the difficulty when performing the shape operation with the robot.In the robot motion generation phase, the robot motion that realizes the shape operation determined in the previous phase is generated.Based on the string shape data acquired in the planning of the shape operation phase, the hand trajectory, string gripping position, and release position are determined.The generated robot motion is executed, and the shape operation is performed in the final phase.

Shape abstraction of string
In this section, shape abstraction data of a string named P-data will be introduced.P-data are generated from a point chain model to describe the topological state of a string.In addition, Reidemeister move and Cross are adopted as a basic unit of shape operations in this study.It was demonstrated that any topological state of a string can be realized by combining Reidemeister move and Cross.The details of Reidemeister move and Cross are described in this section.

P-data
In this research, P-data are adopted for the string tying task.P-data are abstract data that describes the topological state of a string.Takamatsu et al. [5] proposed a method for knotting tasks using P-data.P-data are obtained from a projection of a string.As shown in Figure 2(a), an actual string in 3D space can be expressed as a projection in a 2D plane.In a projection, the ends and intersections of a string are defined as endpoint and node, respectively.
As shown in Figure 2(b), one of the two endpoints is defined as the initial endpoint, and the other is defined as the terminal endpoint.Segments are defined as partial lines which have a connection from one node to another node or from a specified endpoint to the neighbour node.Then, segment ID i is numbered in order from the initial endpoint to the terminal endpoint.As noted above, P-data are an abstraction data that describe the topological state of the string represented by the projection drawing in the form of a matrix.
The string projection diagram and its P-data are shown in Figure 3.The first and second lines of P-data are the identification numbers of nodes.The third line is a type of node called Sign.As shown in Figure 3, the direction in which the segment passing below intersections or the segment passing above is represented by + or −.The character on the 4th line represents the upper and lower (Vertical) of the string and takes either the value of U (Upper) or L (Lower).In the 5th line, the value determined by the combination of Sign and the upper and lower (Vertical) of the string (1 : −/ U, 2 : −/ L, 3 : + / U, 4 : + / L) is entered.If the number of intersections is 0, in other words, the string is untied, and P-data are an empty matrix.

Motion primitive
Reidemeister move and Cross are adopted as a basic unit of shape operations in this study.These are defined in knot theory [17].There are three types of Reidemeister move, which are named Reidemeister move I, Reidemeister move II and Reidemeister move III.Reidemeister move I is an operation to make a simple loop as shown in Figure 4(a).Reidemeister move II is an operation to intersect a segment with another segment as shown in Figure 4(b).Reidemeister move III is an operation to move a series of segments across a node as shown in Figure 4(c).Cross is an operation to make an endpoint intersect a segment as shown in Figure 4(d).
There are two types of Cross, which are Cross of the initial endpoint and Cross of the terminal endpoint.As shown in Figure 4, Reidemeister move and Cross include both operations to add nodes and operations to reduce nodes.
It was demonstrated that any topological state of a string can be realized by combining Reidemeister move and Cross.However, in this paper, we aim to realize the string tying operation using only Reidemeister move I and Cross, based on the following reasons.
Firstly, the string tying operation can be realized even just using Reidemeister move I and Cross.Secondly, if the operations of Reidemeister move II and Reidemeister move III are used, more operation options can be provided when performing the tying operation at each step.However, as the operation steps increase, the tree diagram representing the knot state transition will not only increase in depth but also become larger horizontally, which will increase the total time to decide the next operation.So, Reidemeister move II and Reidemeister move III are not used in the current planning method.Therefore, Reidemeister move I, Cross of the initial endpoint, and Cross of the terminal endpoint are defined as basic shape operations in this paper.

Reidemeister move I
Reidemeister move I (RM I) in shape operations is an operation to make a simple loop described as R I+ or release a simple loop described as R I− , so the number of nodes increases when R I+ is performed and the number of nodes decreases when R I− is performed.Since there are two types of nodes, + and − defined as s in Equation ( 1), there are two possible operations: creating a + node and creating a − node.
Also, the operation of creating the same type of node can be divided into the case of creating a loop on the right side of the segment and the case of creating it on the left side.Here, the parameter d(d = r: right side, d = l: left side) is defined according to whether the created loop is on the right side or the left side of the node when looking towards the terminal endpoint.The parameter n means the number of the segment to be operated.When the type of node to be created is s, RM I is described as Next, Figure 5(a) shows the case when RM I is performed on the segment in contact with the outer face.As shown in Figure 5(a), F 1 , called face or inner face, represents the internal regions bounded by a closed curve, but the unbounded face surrounding the projection drawing is called the outer face, defined as F O .
In Figure 5(a), the same R I+ (2, l, −) operation is performed, but two different shapes are obtained, and the shape operation on the right side of the figure only creates a simple loop.On the other hand, the shape operation on the left side has a structure in which the circumference is surrounded by a loop created by RM I.A closer look reveals that these two types of shapes can be distinguished by whether the direction of the segment is clockwise or counterclockwise.Therefore, by introducing a new parameter t(t = cw: clockwise, t = ccw: counterclockwise) that indicates whether the direction of the segment is clockwise or counterclockwise, RM I is described as Therefore, the shape operation on the left side of the figure is R I+ (2, l, −, cw), and the shape operation on the right side is R I+ (2, l, −, ccw).When not in contact with the outer face, RM I can be classified into four types, as shown in Figure 5(b).

Cross
Cross is an operation in which an endpoint intersects a segment.Since there are two types of endpoints, the initial endpoint and the terminal endpoint, there are always two types of Cross: Cross of the initial endpoint and Cross of the terminal endpoint.The first thing that must be considered as a parameter for shape operation by Cross is the number of the segment crossed by the endpoint.In addition, it must be considered whether the segment crosses from the right side to the left side or from the left side to the right side and whether the endpoint crosses the upper side or the lower side of the segment.
Here, the number n means the segment crossed by the endpoint.The parameter d(d = r: right side, d = l: left side) indicates the direction the endpoint crosses the segment.The parameter v (v = U: upper side, v = L: lower side) indicates whether the endpoint crosses the upper side or the lower side of the segment.Cross of the initial endpoint is described as and Cross of the terminal endpoint is described as However, as shown in Figure 6(a), the shape operation by Cross is performed when the endpoint belongs to the outer face.In Figure 6(a), the same C I+ (2, 1, U) shape operation is performed, but two different shapes may be obtained.As with RM I, it can be seen that the segment can be distinguished by whether it orbits the outer circumference clockwise or counterclockwise.Therefore, by introducing a new parameter t(t = cw: clockwise, t = ccw: counterclockwise) that indicates whether the direction of the segment is clockwise or counterclockwise, Cross of the initial endpoint on the outer face is described as and Cross of the terminal endpoint is described as

Problems with P-data in string knot operation
We considered a plan for string knot operation using Pdata.We concluded that P-data does not have enough information to express the topology of a string in the tying operation.As shown in Figure 7, there is an example of generating an overhand knot from an entirely untied string.When applying R I+ (1, l, −, ccw) and C I+ (2, r, L, ccw) to the string, the knot A is generated in a correct route.The knot A is generated when R I+ (1, r, +, cw) and C I+ (2, l, L, cw) are applied sequentially.However, both of the P-data are the same.Furthermore, the knot B and the knot B also have the same P-data.In other words, it can be said that the difference between knot A and A and B and B cannot be distinguished by the operation planning method using only P-data, and the shape operation that generates the knot B of the target shape cannot be correctly planned.Let us consider the reason why the same P-data has different shapes.Figure 8 shows the knot that can be represented by the P-data of the knot B in Figure 7. Here, the arrows indicate the direction from the initial endpoint to the terminal endpoint.The numbers in the figure are the numbers of the intersections ordered from the initial endpoint.Figure 8(a) shows all connections except the connection between intersection (3) and intersection (4). Figure 8(b) shows the shape when the connection between intersection (3) and intersection (4) is realized by passing the upper side of the entire knot in addition to the state of Figure 8(a), and Figure 8(c) shows the shape in addition to the state of Figure 8(a).It is the shape when it is realized by passing through the lower side of the whole.
In this way, even with the same P-data, there are two options for the node that passes outside the entire string on the projection drawing.It is not known under what conditions this will occur.In any case, it was found that the topological state information of the entire string cannot be expressed by P-data expressed by the information of the intersection in this way.To solve this problem, the Face-list proposed in this study identifies the difference between the knots A and A , B and B by incorporating information on faces that are areas surrounded by several segments.P-data can be used with Face-list to plan shape operations correctly.

Proposal of Face-List
In this section, a new shape abstraction data of string named Face-list proposed in this paper will be introduced.Whereas P-data consisted only of information on the intersections of string projections, Face-list consists of information on faces surrounded by several segments.We aim to realize the string tying operation effectively by complementing information that cannot be obtained from P-data with Face-list.

Overview of Face-List
As shown in Figure 9(a), a face is a region surrounded by a closed curve composed of several segments, and the Face-list is data that lists multiple faces included in the string projection drawing in a list format.Face-list is composed of segment information, while P-data are composed of intersection information.
The segment becomes directional as one of the two endpoints is defined as the initial endpoint and the other as the terminal endpoint in the string projection drawing.The direction from the initial endpoint to the terminal endpoint is defined as the forward direction, and the direction from the terminal endpoint to the initial endpoint is defined as the back direction.
The faces are described using segment numbers and directions ("f " means forward direction and "b" means back direction).The face F 1 shown in Figure 9    and the face F 2 are described as The directional segment number sequence starts with the smallest number, and the closed curves are listed in the order of counterclockwise construction.F 1 and F 2 are internal regions bounded by a closed curve, but the unbounded face surrounding the projection drawing is called the outer face.Therefore, F 1 and F 2 are also called inner faces.The outer face F O shown in Figure 9(b) is bounded to the inner face by a clockwise closed curve consisting of S 3 in the back direction and S 4 in the back direction and contains the initial endpoint inside.Since the segment S 1 including the initial endpoint is located between S 3 and S 4 , it is described as When these three faces are listed in a list format, it becomes Equation ( 9), defined as Face-list.
The numbers on the inner face are sorted in the natural order with respect to the segment number sequence.The outer face is at the end of the list, and in the case of the string projection diagram shown in Figure 9(b), the outer face F O is the third face F 3 .

Merit of the proposed method
Figure 10 shows the transition of the string shape when the overhand knot is generated from an entirely untied string by combining P-data and Face-list.When using only P-data, knots A and A , B and B could not be distinguished because P-data are the same, but these knots can be distinguished by using the face list together.This means that the computer can distinguish the difference between knot A and A and can correctly plan the shape manipulation until knot B is generated.
Figure 10.Representation of string shape transition using P-data and Face-list.
can be passed.False means that the segment in this direction has been passed and cannot be passed.Next, it is possible to create a list of segments that have passed until one round, starting from the forward of the segment with the smaller number in the above segment non-passed table, as shown in Table 1.The segment that is passed in the direction Forward or Back will be set as False in the segment non-passed table in the passing direction.In addition, as shown in Figure 12(a), when the node is encountered while tracing the segment, the search is performed by changing the course to the left with respect to the travelling direction.In other words, the rules can be described as follows: • When reaching the intersection from the N 1 i segment, proceed in the direction of the N 2 i segment; • When reaching the intersection from the N 2 i segment, proceed in the direction of the N 3 i segment; • When reaching the intersection from the N 3 i segment, proceed in the direction of the N 4 i segment; • When reaching the intersection from the N 4 i segment, proceed in the direction of the N 1 i segment.
When the endpoint is reached, it wraps around and proceeds, as shown in Figure 12(   but it is not possible to determine which of the detected faces corresponds to the outer face.Therefore, the outer face is determined by calculating the area of the face, and a Face-List is generated.Since the point chain model consists of multiple points and a straight-line link connecting them, the plane included in the projection drawing of the point chain model is a polygon, as shown in Figure 13.If the coordinates of each vertex of the polygon are q 1 , • • • , q i = [x i , y i ] T , • • • , q v , the area S can be calculated by the following Equation (10).
Because the polygon has a closed-loop form, q v+1 equals q 1 .As a feature of this formula, the area is calculated with a positive value when the arrangement of vertices is counterclockwise, and the area is calculated with a negative value when it is clockwise.In other words, it can be said that the face whose area is calculated as a negative value is the outer face.After determining the outer face, each face is sorted in the natural order with respect to the registered segment number.

Automatic selection of optimal shape operation
Based on the above-described method and the deformation law of P-data and Face-list by the basic shape operation, a combination of the basic shape operation that changes the current string shape to the target shape can be derived.
Theoretically, the target shape can be generated by applying any of these combinations.In an actual environment using a manipulator, the probability that the operation will be successful varies depending on some environmental factors, such as the type of basic shape operation to be performed, the shape of the string, and the physical characteristics.
Therefore, the difficulty of the shape operation to be applied is scored as a cost.Furthermore, by introducing cost calculation into the tree search algorithm and selecting the combination of basic shape operations with the smallest total cost as the optimum operation, the possibility of successful shape operation by the manipulator is increased.RMI + (in) is an operation that creates a simple loop and corresponds to the four types of operations shown in Figure 5(b).RMI + (out) is an operation to create a shape in which the entire string is surrounded by the created loop when creating a loop on the outer face, as shown in Figure 14.The operations of R I+ (n, r, * , ccw) and R I+ (n, r, * , cw) are applicable.RMI + (in) has the lowest cost because it has less interference with the segment and the robot hand, and the success rate was high in the trials by preliminary experiments.On the other hand, in the operation of RMI + (out), it is necessary to arrange the created loop to surround the entire string, and the success rate of the operation is extremely low because it is accompanied by a large deformation of the string.Therefore, the cost is set to the highest.
Next, the cost related to Cross will be described.CR + (U, in) and CR + (L, in) are operations that cross the endpoint belonging to the inner face to the segment, CR + (U, out) and CR + (L, out) are operations to cross the endpoints belonging to the outer face to the segment, U and L represent the upper side or the lower side of the intersecting segment.
Since CR + (U, in) moves the endpoint belonging to the inner face, the deformation of the string is small, and since the endpoint intersects on the segment, there is little interference between the segment and the robot hand, and the success rate is high.The cost is set relatively low.
In the case of CR + (L, in), the deformation of the string is also small, but the cost is higher than the former because the contact between the segments is expected when the endpoint cross under the segment.
CR + (U, out) is an operation to cross the endpoint belonging to the outer face with the segment, so the deformation of the string is large, but since the endpoint intersects on the segment, there is little interference between the segment and the robot hand.
In CR + (L, out), in addition to a large string deformation, the endpoint intersects under the segment, so interference between segments or between the segment and the robot hand is also possible.Therefore, the highest cost is set among Cross in the string tying operation.
The costs shown in Table 2 have been determined for the above reasons, but these costs are related to the current robot system and vary depending on the measurement accuracy of the sensor and the hand accuracy of the robot.It also needs to be reviewed by changing the motion generation method of motion primitive such as Reidemeister move I and Cross.The optimum combination is obtained by the above cost calculation from the combinations of multiple shape operations derived.The first basic shape operation in the combination of the optimum shape operations is determined as the optimum shape operation, and the operation by the manipulator is executed.For example, Figure 17 shows the state transition of the knot that generates the overhand knot in a tree diagram with cost.It is a part of the tree diagram, and the choices increase as the depth increases from the initial shape in the calculation.In this case, the combination of the basic shape operations R I+ (1, l, −, ccw) → C T+ (2, r, U, ccw) → C T+ (3, l, L) indicated by the thick arrow shows the lowest total cost, and its value is 17.Then, the first shape operation, R I+ (1, l, −, ccw), is selected as the optimum shape operation.
In this paper, breadth-first search is adopted.The reason to use breadth-first search is that depth-first search cannot be adopted, because there is no limit to the depth when the original state of the string is entirely untied.Since the current target shape is simple, although no problem has occurred in the breadth-first search, it is necessary to suppress the increase in search time for more complicated shape operations.Pruning the tree structure and improving the search method are future tasks.

Generation of the motion of a manipulator
In this subsection, the conversion from abstract primitive motion to concrete robot motions will be introduced.To describe the robot's motion, we have defined the following functions that calculate the coordinates and angles of the points within the segment of the point chain model.Figure 15 shows some points around a focused intersection in segments of point chain model.The j-th segment (j = 1, 2, • • • , N) of point chain model from the initial endpoint is described as jth segment.j P i = [ j P xi , j P yi , j P zi ] T (i = 1, 2, • • • , n j ) is defined as a 3D vector which represents the i-th point of j-th segment from j-th segment's end which is closer to the initial endpoint.
Figure 16 shows the pose of the hand in a grasping motion.The k-th desired gripper position of the motion trajectory is defined as H k , which is a 3D vector.Angle of the hand and gripper's state at a desired gripper position H k are defined as θ k and G k respectively.In addition, a function A( j P i ) to calculate an angle and a function P m (j) to determine the mean point of j-th segment are defined as Equation (1) and Equation (2), respectively.
A( j P i ) = tan −1 j P y(i+1) − j P yi j P x(i+1) − j P xi (11) P m (j) = ⎧ ⎨ ⎩ j P n j includes the endpoint to be manipulated and intersecting it with another segment from below.
Here, σ represents the total number of segments, and γ represents the segment number that includes the endpoint to be manipulated, and P e represents the coordinate of the endpoint to be manipulated.When the type of shape manipulation is Cross C I+ at the initial endpoint, the value of γ is determined to be 1, and P e is determined to be 1 P 1 .On the other hand, when it is Cross C T+ at the terminal endpoint, the value of γ is determined to be σ , and P e is determined to be σ P n σ .Furthermore, the distance l between the endpoint to be manipulated and the middle point of the intersecting segment is calculated with l = |P e − P m (γ )|.
The shape operation of CR + (L, in) is performed in two steps according to the value of l.In the case of l ≤ 0.05[m], the first step operation is performed to bring the endpoint closer to the intersecting segment, and the details of each desired gripper position are shown in Table 4.The vector a (a = P m (γ ) − γ P 2 ) is an offset to keep the endpoint and the intersecting segment at a constant distance.
In the case of l > 0.05[m], the second step operation is performed to lift up the intersecting segment and move it over the segment including the endpoint, and the details of each desired gripper position are shown in Table 5.

String tying experiment
The experiments of the string tying operation were conducted to evaluate the proposed method.In the Close experiments, the target shape is the overhand knot shown in Figure 17.P-data and Face-list of the overhand knot are also described in this figure.The experiments were conducted with two initial shapes: an entirely untied string shown in Figure 17 and a shape similar to the overhand knot shown in Figure 2. Case 1 is the experiment to generate the overhand knot from the initial shape of the former, and Case 2 is the experiment from the initial shape of the latter.The shape of the latter is the same as the overhand knot at first glance, but it is not the overhand knot because the top and bottom of the upper right intersection are different from the overhand knot.The details of the experiments and the results of the experiments are described in this section.
Figure 18 shows the environment of the experiments and set of coordinate systems.It is an industrial manipulator RV6SL manufactured by Mitsubishi Electric Corporation, and a gripper and a range camera Creative Senz3D are attached to the manipulator's hand.
The world coordinate system W is fixed at a position 700 [mm] away from the coordinate system 0 fixed to the base part of the manipulator.Also, e is the coordinate system fixed to the tip of the gripper,   and d is the coordinate system of the range camera for measuring the shape of the string.
Figure 19 shows the state of the experiment of Case 1.The shape operation was performed three times until the target shape was reached.After each shape operation, the distance camera is used to recognize the current string shape, and the tree diagram of the string shape operation plan is calculated to select the robot motion plan.
The total work time is about 12 minutes, and it mainly takes time to generate the tree diagram when planning the shape operation of the string.Reducing the calculation time is a future task.Figure 16(  Figure 20 shows the state of the experiment of Case 2. In the first loop, one intersection is removed by the CR − (U, U) operation.In the second loop, the CR + (L, in) operation is performed.the CR + (L, in) operation is divided into two steps.In the first step, the segment with the endpoint is grasped and brought closer to the intersecting segment.In the second step, the segment to intersect is grasped and intersects the endpoint.In the experiment of Case 2, the intended deformation is performed, and the operation is successful up to the target shape.
In this research, the current state of the string is acquired by using a distance camera for each operation, and the robot motion planning is performed.Although the explanation of shape recognition is omitted in this paper, the shape recognition algorithm is detailed in reference [16].The results of the experiments were based on automatic calculations by a programme that implemented the proposed method, and no human intervention was involved during the experiments.According to the results, we can confirm the usefulness of the Face-list operation planning method proposed in this study.

Conclusions
In this paper, we proposed an operation planning method using a new shape abstraction data Face-list to execute the string tying operation effectively.We described the problems of P-data, which is existing abstraction data, and proposed Face-list as complementary information.Finally, the robot motion plan based on the sensor information was generated using the proposed description method, and its effectiveness was confirmed by the experiments using a manipulator.Future issues will be described.In our research approach, primitive motion such as Cross, which is linked to the generated tree diagrams, has been executed one by one for the target shape.However, it is possible to reduce the probability of work failure and shorten the work time by executing multiple consecutive primitive motion as humans do because the number of switching times is reduced.To realize such a continuous primitive motion, an algorithm that predicts shape deformation due to interference between strings is required.
Finally, we describe the operation of more complicated shapes.To knot a more complicated knot, an operation plan that takes into account the length of the knot element is required.It is possible to generate an operation plan for any complex knot in a topological space using P-data and Face-list.However, since the length of the actual string is finite, if the distance between each intersection and both ends is not appropriate, it will fail when forming a knot.It is considered that this is because when a large curvature is given to a short segment, the reaction moment becomes large and unintended deformation is likely to occur with respect to the shape operation.In order to solve this problem, it is necessary to select the position of the intersection on the string in consideration of the influence on the final shape when forming the initial intersection.

Figure 1 .
Figure 1.Overview of the string tying operation: (a) Shape recognition (b) Output of point cloud data (c) Generation of point chain model (d) Planning of shape operation (e) Robot motion generation (f) Executing the operation up to the target shape.

Figure 2 .
Figure 2. (a) Projection of a string, (b) Projection of a string with numbered segments.

Figure 3 .
Figure 3.A projection of string and its P-data.

Figure 4 .
Figure 4. Basic shape operations by Reidemeister move and Cross.

Figure 5 .
Figure 5. (a) Shape operation by RM I in outer face.(b) Four types of shape operations by RM I.

Figure 6 .
Figure 6.(a).Shape operation by RM I in outer face.(b) Four types of shape operations by Cross.
(b) is surrounded by a closed curve in the counterclockwise direction consisting of S 2 in the forward direction and S 3 in the forward direction, and the terminal endpoint is included in the plane.The face F 2 shown in Figure9(b) is surrounded by a closed curve in the counterclockwise direction consisting of S 2 in the back direction and S 4 in the forward direction.The face F 1

Figure 7 .
Figure 7. Representation of string shape transition using P-data.

Figure 8 .
Figure 8. Two possibilities to connect pair of intersection through outside of whole string in projected string shape.

Figure 9 .
Figure 9. Faces in a projection of string: (a) Definition of face, (b) Numbered Segments and faces.

Figure 11 .
Figure 11.A projection of string with numbered nodes.

Figure 12 .
Figure 12.Overview of detecting faces: (a) the Rules of detecting faces, (b) the Results of detecting faces.

Figure 13 .
Figure 13.A face in projection of point chain model.

Figure 14 .
Figure 14.An example of string deformation by RMI + (out).

Figure 15 .
Figure 15.Points in segments of point chain model.

Figure 16 .
Figure 16.Pose of the hand in grasping motion.

Figure 17 .
Figure 17.Tree diagram representing knot state transition to overhand knot with costs.

Figure 18 .
Figure 18.The environment of string untying experiment and coordinate systems.
a) is the first shape operation.The motion to manipulate the string with Reidemeister move I is performed and succeeded, as shown in Figure 19(b).Next, an overhand knot is generated by the two times Cross operations shown in Figure 19(c) and (d).
Figure 19(e,f) shows the shape of the string created by three operations.

Figure 20 .
Figure 20.The experiment result of case 2.

Table 2 .
The evaluation and cost both of Cross and Reidemeister Move in the string tying operation.

Table 4 .
Robot motion of the first step of CR + (L, in) at each desired gripper position k.

Table 5 .
Robot motion of the second step of CR + (L, in) at each desired gripper position k.