US6513046B1
Expired / Lapsed
US
⏱️ 0.0 years remaining
Storing and recalling information to augment human memories
Assignee: Microsoft Technology Licensing LLC
•
Filing: 1999-12-15
•
Granted: 2003-01-28
•
Expires: 2019-12-15
Abstract
Abstract A system for computer-based storing of information about a current state so that later recall of the information can augment human memories. In particular, when information about a current event of interest is to be stored, a variety of current state information of different types (e.g., video, audio, and textual information) about the environment and about a user can be acquired via sensors and other input devices. The variety of state information can then be associated together as a group and stored for later retrieval. Other information can also be associated with the group, such as one or more recall tags that facilitate later retrieval of the group, or one or more annotations to provide contextual information when the other state information is later retrieved and presented to the user. When information about a past event is to be recalled, one or more identifying recall tags can be received that are used to identify one or more state information groups that match the identifying tags. Some or all of the previously-acquired state information for the identified state information groups can then be presented to the user on appropriate output devices. Other information, such as annotations, can also be presented to the user in order to describe the state information and thus assist the user's recollection of the previous state when the information was stored.
Inventors
- 👤 Kenneth H. Abbott, III
- 👤 Dan Newell
- 👤 James O. Robarts
Bibliographic Details
Application Number
—
Jurisdiction
US
Data Source
google_patents
Last Synchronized
2026-10-24 16:25
CPC Classifications
G; G06; G06F; G06F16/00; G06F16/90; G06F16/907; G06F16/40; G06F16/48; G06F16/487; Y; Y10; Y10S; Y10S707/00; Y10S707/99941; Y10S707/99944; Y10S707/99945; Y10S707/99948
Claims Analysis
Total Claims: 58
Claim 1
Independent Claim
397 words
A method in a computer to assist a user in recalling a previous environment of the user, the computer having input devices capable of recording audio and video information from the environment and having output devices capable of presenting audio and video information, the method comprising: receiving a plurality of indications from the user, each received indication to store a state fragment having information about a current state of an environment of the user, the indications received by a State Storage Module executing on the computer; for each of the plurality of received indications, storing a state fragment by, recording video information from the environment using at least one of the input devices; recording at least one audio recall tag from the user using at least one of the input devices, the recall tag related to a subject of the recorded video information, the recording performed under control of the State Storage Module; recording at least one audio annotation from the user using at least one of the input devices, the annotation providing descriptive information about the recorded video information, the recording performed under control of the State Storage Module; and creating a stored state fragment having the recorded video information, the recorded audio recall tags, and the recorded audio annotations, the creating performed by the State Storage Module; and when the user indicates to present stored information from one of the stored state fragments in order to assist the user in recalling the environment of the user at the time of the storing of that state fragment, recalling a stored state fragment by, receiving from the user an indication of a subject, the indication of the subject received by a State Recall Module executing on the computer; comparing the indicated subject to the recall tags of the stored state fragments, the comparing performed by the State Recall Module; and when the indicated subject matches a recall tag of one of the stored state fragments, under control of the State Recall Module, presenting to the user on at least one of the output devices the recorded video information of the one stored state fragment; and presenting to the user on at least one of the output devices the recorded audio annotations of the one stored state fragment, so that the user can recall video and annotation information for a previous state of an environment of the user.
Claim 2
Independent Claim
397 words
A method in a computer to assist a user in recalling a previous environment of the user, the computer having input devices capable of recording audio and video information from the environment and having output devices capable of presenting audio and video information, the method comprising: receiving a plurality of indications from the user, each received indication to store a state fragment having information about a current state of an environment of the user, the indications received by a State Storage Module executing on the computer; for each of the plurality of received indications, storing a state fragment by, recording video information from the environment using at least one of the input devices; recording at least one audio recall tag from the user using at least one of the input devices, the recall tag related to a subject of the recorded video information, the recording performed under control of the State Storage Module; recording at least one audio annotation from the user using at least one of the input devices, the annotation providing descriptive information about the recorded video information, the recording performed under control of the State Storage Module; and creating a stored state fragment having the recorded video information, the recorded audio recall tags, and the recorded audio annotations, the creating performed by the State Storage Module; and when the user indicates to present stored information from one of the stored state fragments in order to assist the user in recalling the environment of the user at the time of the storing of that state fragment, recalling a stored state fragment by, receiving from the user an indication of a subject, the indication of the subject received by a State Recall Module executing on the computer; comparing the indicated subject to the recall tags of the stored state fragments, the comparing performed by the State Recall Module; and when the indicated subject matches a recall tag of one of the stored state fragments, under control of the State Recall Module, presenting to the user on at least one of the output devices the recorded video information of the one stored state fragment; and presenting to the user on at least one of the output devices the recorded audio annotations of the one stored state fragment, so that the user can recall video and annotation information for a previous state of an environment of the user.
Claim 3
Dependent Claim
45 words
The method of claim 1 wherein the indicated subject matches a recall tag of multiple of the stored state fragments, and wherein the presenting of the recorded video information and of the recorded audio annotation is performed for each of the multiple stored state fragments.
Claim 4
Dependent Claim
48 words
The method of claim 1 wherein the indicated subject matches a recall tag of multiple of the stored state fragments, and wherein the presenting of the recorded video information and of the recorded audio annotation is performed for a stored state fragment having a highest degree of matching.
Claim 5
Dependent Claim
83 words
The method of claim 1 including: for each of a plurality of indications generated automatically by the computer without human intervention, storing a state fragment by, recording video information from the environment using at least one of the input devices; receiving a computer-generated recall tag related to a subject of the recorded video information; receiving a computer-generated annotation providing descriptive information about the recorded video information; and creating a stored state fragment having the recorded video information, received recall tag, and received annotation.
Claim 6
Dependent Claim
120 words
The method of claim 1 including: when the computer automatically indicates without human intervention that one of the stored state fragments is to be recalled, recalling a stored state fragment by, receiving a computer-generated indication of a subject; comparing the indicated subject to the recall tags of the stored state fragments; and when the indicated subject matches a recall tag of one of the stored state fragments, presenting to the user on at least one of the output devices the recorded video information of the stored state fragment having the matching recall tag; and presenting to the user on at least one of the output devices the recorded audio annotations of the stored state fragment having the matching recall tag.
Claim 7
Dependent Claim
19 words
The method of claim 1 including deleting the one stored state fragment after the recalling of that state fragment.
Claim 8
Dependent Claim
30 words
The method of claim 1 including, after the storing of the one state fragment and before the recalling of that state fragment, manipulating the information stored in that state fragment.
Claim 9
Dependent Claim
36 words
The method of claim 1 including, after the storing of the state fragments, indexing the stored state fragments based on the recall tags to facilitate later recall of a state fragment matching a specified recall tag.
Claim 10
Dependent Claim
158 words
A computer-implemented method for facilitating later recall by a user of a prior state of an environment external to the user, the method comprising: storing information about a current state of the environment by, recording information from the environment with at least one input device of the computer; receiving an indication of a recall tag to assist in later retrieval of the recorded information from the environment, the indication received by an executing State Storage Module; receiving an indication of descriptive information about the environment the indication received by the State Storage Module; and associating the indicated recall tag and the indicated descriptive information with the recorded environment information, the associating performed under control of the State Storage Module, so that if the recall tag is later supplied, the recorded environment information and the descriptive information can be presented to augment the memory of the user of the state of the environment at the time of the storing.
Claim 11
Dependent Claim
158 words
A computer-implemented method for facilitating later recall by a user of a prior state of an environment external to the user, the method comprising: storing information about a current state of the environment by, recording information from the environment with at least one input device of the computer; receiving an indication of a recall tag to assist in later retrieval of the recorded information from the environment, the indication received by an executing State Storage Module; receiving an indication of descriptive information about the environment the indication received by the State Storage Module; and associating the indicated recall tag and the indicated descriptive information with the recorded environment information, the associating performed under control of the State Storage Module, so that if the recall tag is later supplied, the recorded environment information and the descriptive information can be presented to augment the memory of the user of the state of the environment at the time of the storing.
Claim 12
Dependent Claim
24 words
The method of claim 9 wherein the storing of the information about the current state is in response to an indication from the user.
Claim 13
Dependent Claim
26 words
The method of claim 9 wherein the storing of the information about the current state is in response to an indication generated automatically without human intervention.
Claim 14
Dependent Claim
36 words
The method of claim 9 including, after the storing of the information about the current state, presenting to the user the stored information about the current state in response to a received indication matching the tag.
Claim 15
Dependent Claim
11 words
The method of claim 9 wherein the tag is textual information.
Claim 16
Dependent Claim
11 words
The method of claim 9 wherein the tag is audio information.
Claim 17
Dependent Claim
11 words
The method of claim 9 wherein the tag is video information.
Claim 18
Dependent Claim
23 words
The method of claim 9 wherein the stored information about the current state includes sensed information about a current state of the user.
Claim 19
Dependent Claim
20 words
The method of claim 9 wherein the stored information about the current state includes input information received from the user.
Claim 20
Dependent Claim
22 words
The method of claim 9 wherein the stored information about the current state includes information about a current state of the computer.
Claim 21
Dependent Claim
16 words
The method of claim 9 including storing information about a plurality of states of the environment.
Claim 22
Independent Claim
160 words
A method in a computer for facilitating later recall by a user of a previous state, the method comprising: in response to each of a plurality of indications from the user, storing a group of information about a current state by, receiving from the user an indication of at least one input sensor of the computer, recording information about the current state with the indicated input sensors; receiving from the user an indication of a topic related to the current state to assist in later retrieval, the indication received by a State Storage Module of the computer; receiving from the user an indication of information describing the current state, the indication received by the State Storage Module; and associating the indicated topic and the indicated descriptive information with the recorded information, the associating performed under control of the State Storage Module; and indexing the stored groups of information in order to facilitate later retrieval of a stored group of information.
Claim 23
Independent Claim
160 words
A method in a computer for facilitating later recall by a user of a previous state, the method comprising: in response to each of a plurality of indications from the user, storing a group of information about a current state by, receiving from the user an indication of at least one input sensor of the computer, recording information about the current state with the indicated input sensors; receiving from the user an indication of a topic related to the current state to assist in later retrieval, the indication received by a State Storage Module of the computer; receiving from the user an indication of information describing the current state, the indication received by the State Storage Module; and associating the indicated topic and the indicated descriptive information with the recorded information, the associating performed under control of the State Storage Module; and indexing the stored groups of information in order to facilitate later retrieval of a stored group of information.
Claim 24
Dependent Claim
46 words
The method of claim 20 including: receiving information from the user to identify one of the stored groups of information; identifying the one of the stored groups by using the received information and the indexing; and presenting to the user information from the identified stored group.
Claim 25
Dependent Claim
22 words
The method of claim 20 wherein the recorded information about the current state includes sensed information about an environment of the user.
Claim 26
Dependent Claim
20 words
The method of claim 20 wherein the stored groups of information include information about a current state of the computer.
Claim 27
Independent Claim
214 words
A method in a computer for facilitating later recall by a user of a previous state, the method comprising: monitoring the user to collect a group of information about a current state of the user; monitoring an environment of the user to collect a group of information about a current state of the environment; and without user intervention, determining based on the collected groups of information a level of user interest in at least one of the current state of the user and the current state of the environment, the determining performed by an executing User Modeling Module; and when the level of user interest in at least one of the current states is determined to exceed a threshold, storing information about at least one of the current states by, recording information about at least one of the current states with at least one input sensor of the computer; determining a topic related to the recorded information, the determining performed under control of an executing State Storage Module; and associating the determined topic with the recorded information, the associating performed under control of the State Storage Module, so that the stored information can later be presented to the user to assist the user in recalling the current states at the time of the recording.
Claim 28
Independent Claim
214 words
A method in a computer for facilitating later recall by a user of a previous state, the method comprising: monitoring the user to collect a group of information about a current state of the user; monitoring an environment of the user to collect a group of information about a current state of the environment; and without user intervention, determining based on the collected groups of information a level of user interest in at least one of the current state of the user and the current state of the environment, the determining performed by an executing User Modeling Module; and when the level of user interest in at least one of the current states is determined to exceed a threshold, storing information about at least one of the current states by, recording information about at least one of the current states with at least one input sensor of the computer; determining a topic related to the recorded information, the determining performed under control of an executing State Storage Module; and associating the determined topic with the recorded information, the associating performed under control of the State Storage Module, so that the stored information can later be presented to the user to assist the user in recalling the current states at the time of the recording.
Claim 29
Dependent Claim
27 words
The method of claim 24 including: after the storing of the information, receiving information that identifies the stored information; and presenting to the user the stored information.
Claim 30
Dependent Claim
22 words
The method of claim 24 wherein the determined topic is a recall tag to be later used in identifying the stored information.
Claim 31
Dependent Claim
29 words
The method of claim 24 including, without user intervention, determining information describing the recorded information and associating the determined descriptive information with the determined topic and the recorded information.
Claim 32
Dependent Claim
159 words
A computer-implemented method for facilitating recall by a user of a previous state, the method comprising: storing information about a current state by, recording information from an environment external to the user with at least one input device of the computer; recording information from the user with at least one input device of the computer; receiving an indication of a recall tag related to the current state, the indicated recall tag to be used for later retrieval of the recorded environment information and the recorded user information, the indication received by an executing State Storage Module; and associating the indicated recall tag with the recorded environment information and the recorded user information, the associating performed under control of the State Storage Module, so that if the recall tag is later supplied, the recorded environment information and the recorded user information can be used to augment the memory of the user of the state at the time of the storing.
Claim 33
Dependent Claim
159 words
A computer-implemented method for facilitating recall by a user of a previous state, the method comprising: storing information about a current state by, recording information from an environment external to the user with at least one input device of the computer; recording information from the user with at least one input device of the computer; receiving an indication of a recall tag related to the current state, the indicated recall tag to be used for later retrieval of the recorded environment information and the recorded user information, the indication received by an executing State Storage Module; and associating the indicated recall tag with the recorded environment information and the recorded user information, the associating performed under control of the State Storage Module, so that if the recall tag is later supplied, the recorded environment information and the recorded user information can be used to augment the memory of the user of the state at the time of the storing.
Claim 34
Dependent Claim
18 words
The method of claim 28 wherein the recorded information from the user includes sensed information about the user.
Claim 35
Dependent Claim
18 words
The method of claim 28 wherein the recorded information from the user includes information input by the user.
Claim 36
Dependent Claim
14 words
The method of claim 28 including associating a received annotation with the stored information.
Claim 37
Independent Claim
187 words
A method for assisting a user of a wearable computer in recalling a past state of the user, the wearable computer having a plurality of input devices, the method comprising; receiving a plurality of indications to store information about a current state; for each of the plurality of received indications, storing a state fragment by, recording sensed information about an environment external to the user from one of the input sensor devices, recording sensed information about the user from one of the input sensor devices; and associating the recorded environment information and the recorded user information as the stored state fragment, the associating performed under control of a State Storage Module of the wearable computer; and augmenting the memory of the user of a past state by, receiving an indication of a topic of interest to the user; identifying at least one stored state fragment having information related to the topic of interest, the identifying performed under control of a State Recall Module of the wearable computer; and presenting to the user the recorded environment information and the recorded user information associated with the identified state fragment.
Claim 38
Independent Claim
187 words
A method for assisting a user of a wearable computer in recalling a past state of the user, the wearable computer having a plurality of input devices, the method comprising; receiving a plurality of indications to store information about a current state; for each of the plurality of received indications, storing a state fragment by, recording sensed information about an environment external to the user from one of the input sensor devices, recording sensed information about the user from one of the input sensor devices; and associating the recorded environment information and the recorded user information as the stored state fragment, the associating performed under control of a State Storage Module of the wearable computer; and augmenting the memory of the user of a past state by, receiving an indication of a topic of interest to the user; identifying at least one stored state fragment having information related to the topic of interest, the identifying performed under control of a State Recall Module of the wearable computer; and presenting to the user the recorded environment information and the recorded user information associated with the identified state fragment.
Claim 39
Dependent Claim
142 words
A computer-readable medium whose contents cause a computer to facilitate providing information to a user about an environment external to the user, by performing a method comprising: storing information about a current state of the environment by, recording information from the environment with at least one input device of the computer; receiving an indication of a recall tag related to the environment, the indication received by a State Storage Module of the computer; receiving an indication of descriptive information about the environment, the indication received by the State Storage Module; and associating the indicated recall tag and the indicated descriptive information with the recorded environment information, the associating performed under control of the State Storage Module, so that if the recall tag is later supplied, the recorded environment information and the descriptive information about the environment can be provided to the user.
Claim 40
Dependent Claim
142 words
A computer-readable medium whose contents cause a computer to facilitate providing information to a user about an environment external to the user, by performing a method comprising: storing information about a current state of the environment by, recording information from the environment with at least one input device of the computer; receiving an indication of a recall tag related to the environment, the indication received by a State Storage Module of the computer; receiving an indication of descriptive information about the environment, the indication received by the State Storage Module; and associating the indicated recall tag and the indicated descriptive information with the recorded environment information, the associating performed under control of the State Storage Module, so that if the recall tag is later supplied, the recorded environment information and the descriptive information about the environment can be provided to the user.
Claim 41
Dependent Claim
12 words
The computer-readable medium of claim 33 wherein the tag is audio information.
Claim 42
Dependent Claim
12 words
The computer-readable medium of claim 33 wherein the tag is video information.
Claim 43
Dependent Claim
24 words
The computer-readable medium of claim 33 wherein the stored information about the current state includes sensed information about a current state of the user.
Claim 44
Dependent Claim
23 words
The computer-readable medium of claim 33 wherein the stored information about the current state includes information about a current state of the computer.
Claim 45
Dependent Claim
121 words
A computer system for reminding a user of a prior environment external to the user, comprising: a state storage module that stores information about a current state of the environment by recording information from the environment with at least one input device of the computer system, by receiving an indication of a recall tag related to the environment for use in later retrieval of the recorded information, by receiving an indication of descriptive information about the environment, and by associating the indicated recall tag and the indicated descriptive information with the recorded environment information; and a storage means for storing the information about the current state of the environment for later providing to a user after receipt of the recall tag.
Claim 46
Dependent Claim
121 words
A computer system for reminding a user of a prior environment external to the user, comprising: a state storage module that stores information about a current state of the environment by recording information from the environment with at least one input device of the computer system, by receiving an indication of a recall tag related to the environment for use in later retrieval of the recorded information, by receiving an indication of descriptive information about the environment, and by associating the indicated recall tag and the indicated descriptive information with the recorded environment information; and a storage means for storing the information about the current state of the environment for later providing to a user after receipt of the recall tag.
Claim 47
Dependent Claim
41 words
The computer system of claim 38 including a state recall module that receives information from the user to identify one of the stored groups of information, and that presents to the user information from the identified one of the stored groups.
Claim 48
Dependent Claim
49 words
The computer system of claim 38 including a user modeling module that monitors the user and an environment of the user to collect a group of information about a current state, and that determines based on the monitoring when to store information about the current state of the environment.
Claim 49
Dependent Claim
144 words
A data structure stored in memory of a computer for use in facilitating recall by a user of a previous state of the user at a previous time, the data structure including a group of associated information that describes the previous state by comprising: stored information about an environment external to the user that was sensed by an input sensor device of the computer at the previous time; stored information from the user that was received by an input device of the computer at the previous time; and a recall tag from a State Storage Module for identifying the stored environment information and the stored user information that was received, so that if tile recall tag is supplied, the stored environment information and the stored user information can be provided to the user to facilitate recall of the previous state at the previous time.
Claim 50
Dependent Claim
144 words
A data structure stored in memory of a computer for use in facilitating recall by a user of a previous state of the user at a previous time, the data structure including a group of associated information that describes the previous state by comprising: stored information about an environment external to the user that was sensed by an input sensor device of the computer at the previous time; stored information from the user that was received by an input device of the computer at the previous time; and a recall tag from a State Storage Module for identifying the stored environment information and the stored user information that was received, so that if tile recall tag is supplied, the stored environment information and the stored user information can be provided to the user to facilitate recall of the previous state at the previous time.
Claim 51
Dependent Claim
18 words
The data structure of claim 41 including an annotation that describes the environment information and the user information.
Claim 52
Dependent Claim
18 words
The data structure of claim 41 including information about a state of the computer at the previous time.
Claim 53
Dependent Claim
20 words
The data structure of claim 41 wherein the user information includes sensed information about the user at the previous time.
Claim 54
Dependent Claim
20 words
The data structure of claim 41 wherein the user information includes information input by the user at the previous time.
Claim 55
Dependent Claim
148 words
A generated data signal transmitted via a data transmission medium, the generated data signal having encoded contents that cause a computing device to assist a user in recalling a previous state at a previous time, the encoded contents of the data signal including a group of associated information that describes the previous state by comprising: information about an environment external to the user that was sensed by an input sensor device of the computer at the previous time; information from the user that was received by an input device of the computer at the previous time; and a recall tag from a State Storage Module for identifying the environment information and the information from the user, so that if the recall tag is supplied, the environment information and the user information can be presented to the user to assist in recalling the previous state at the previous time.
Claim 56
Dependent Claim
148 words
A generated data signal transmitted via a data transmission medium, the generated data signal having encoded contents that cause a computing device to assist a user in recalling a previous state at a previous time, the encoded contents of the data signal including a group of associated information that describes the previous state by comprising: information about an environment external to the user that was sensed by an input sensor device of the computer at the previous time; information from the user that was received by an input device of the computer at the previous time; and a recall tag from a State Storage Module for identifying the environment information and the information from the user, so that if the recall tag is supplied, the environment information and the user information can be presented to the user to assist in recalling the previous state at the previous time.
Claim 57
Dependent Claim
18 words
The data signal of claim 46 including an annotation that describes the environment information and the user information.
Claim 58
Dependent Claim
18 words
The data signal of claim 46 including information about a state of the computer at the previous time.
References Cited (Backward Citations)
US5227614A
Cited
US4916441A
Cited
US5201034A
Cited
US5506580A
Cited
WO1990008361A1
Cited
US5032083A
Cited
US5214757A
Cited
US5983335A
Cited
US5781913A
Cited
US5568645A
Cited
US5208449A
Cited
JPH05260188A
Cited
US6108197A
Cited
US5335276A
Cited
US5416730A
Cited
US5493692A
Cited
US5555376A
Cited
US5611050A
Cited
US5812865A
Cited
US6064943A
Cited
US5470233A
Cited
EP0823813A2
Cited
US5646629A
Cited
US5559520A
Cited
US5959611A
Cited
US5642303A
Cited
US5873070A
Cited
WO1997003434A1
Cited
JPH0991112A
Cited
US5878274A
Cited
US5726660A
Cited
US5948041A
Cited
US5910799A
Cited
US5790974A
Cited
US6047301A
Cited
US5719744A
Cited
US5938721A
Cited
US5902347A
Cited
US5798733A
Cited
US5991687A
Cited
US6294953B1
Cited
USPTO Annuity & Maintenance Windows
| Annuity Stage | Window Opens | Payment Deadline | Surcharge Period Until | Estimated Fee | Status |
|---|---|---|---|---|---|
| 3.5 Years (Year 4) | 2006-01-27 | 2006-07-29 | 2007-01-28 | $2000.00 | Window Closed / Lapsed |
| 7.5 Years (Year 8) | 2010-01-27 | 2010-07-29 | 2011-01-28 | $3760.00 | Window Closed / Lapsed |
| 11.5 Years (Year 12) | 2014-01-27 | 2014-07-29 | 2015-01-28 | $7700.00 | Window Closed / Lapsed |